# Understanding Professional Needs to Create Privacy-Preserving and Secure
Emergent Digital Artworks

Kathryn Lichlyter
University of Denver  Urvashi Kishnani
University of Denver  Kate Hollenbach
University of Denver  Sanchari Das
University of Denver

###### Abstract

In recent years, immersive art installations featuring interactive artworks
have been on the rise. These installations are an integral part of museums and
art centers like selfie museums, teamLab Borderless, ARTECHOUSE, and Meow
Wolf. Moreover, immersive art have also been increasingly incorporated into
traditional museums as well. However, immersive art requires active user
participation and often captures information from viewers and participants
through cameras, sensors, microphones, embodied interaction devices,
surveillance, and kinetic mirrors. Therefore, we propose a new line of
research to examine the security and privacy postures of immersive artworks.
In our pilot study, we conducted a semi-structured interview with five
experienced practitioners from either the art ($2$) or cybersecurity ($3$)
fields. Our aim was to understand their current security and privacy
practices, along with their needs when it comes to immersive art. From their
responses, we created a list of security and privacy parameters, such as,
providing opt-in mechanics for data collection, knowledge of data collection
tools such as proximity sensors, and creating security awareness amongst
participants by communicating security protocols and threat models. These
parameters allow us to build privacy-preserving, secure, and accessible
software for individuals working in media arts, who often have no background
on security and privacy. In the future, we plan to utilize these parameters to
develop software in response to those needs and then host an art exhibition of
immersive artworks utilizing the platform.

Copyright is held by the author/owner. Permission to make digital or hard
copies of all or part of this work for personal or classroom use is granted
without fee.

_9th Workshop on Inclusive Privacy and Security (WIPS)
In Association with USENIX Symposium on Usable Privacy and Security (SOUPS)_
2024\.
August 9, 2024, Virtual.

##  1 Introduction

Immersive art installations represent a unique genre of artistic expression,
characterized by audience engagement and interactivity. In the year $2019$,
the immersive art industry was valued at $61.8 billion and has only grown
since [9]. Within these installations, audiences can become integral
components, contributing to the artwork’s dynamic narrative through their
participation. Some installations even incorporate data collected from
participants, seamlessly integrating it into the artistic experience. For
instance, notable examples include installations that scan, interpret, and
provide feedback on various types of participant data, spanning from
physiological responses to environmental interactions. Design I/O [5]
interactive installations utilize depth sensors that can generate 3D models of
surrounding people and objects [16].

The utilization of technology to capture and utilize audience data underscores
the importance of addressing security concerns surrounding data collection and
storage. As immersive art installations continue to evolve in complexity and
scope, ensuring the security and privacy of collected data becomes imperative.
Interactive artworks utilize microphones, cameras, sensors, and other devices
that can collect data on individuals in real-time [4, 3]. Several artworks,
such as those by Golan Levin [6, 7], Kyle McDonald [13], and Lauren McCarthy
[11], operate by utilizing audiovisual recordings of participants. Other
artworks, such as those by Rafael Lozano-Hemmer [8], Teague McDaniel [12], and
Sam Lavigne and Tega Brain [17], utilize personal data collected from
participants. Such data collection practices may be unknown to audiences and
considered a breach of privacy by some.

This study applied our prior proposal [2] and aimed to delve deeper into the
current approaches, practices, and concerns regarding the cybersecurity of
immersive art installations. By interviewing experts from both the
cybersecurity and mixed media artistry industries, we seek to gain insights
into their methodologies for securing data within their respective domains.
Thus, this research endeavors to explore the following research questions:

  * •

RQ1: In what ways do emergent digital art practitioners consider and
prioritize security and privacy in creating immersive artworks?

  * •

RQ2: What technological needs do installation artists have to create privacy-
preserving and secure artworks?

  * •

RQ3: Through the perspectives of prior research, expert recommendations, and
audience viewpoints, what are the primary privacy and cybersecurity concerns
associated with immersive digital arts?

By synthesizing insights from cybersecurity and media arts experts, as well as
considering audience perspectives, this research aims to elucidate the major
themes, concerns, and recommendations regarding security and privacy of
individuals immersed in interactive art installations.

##  2 Method

To answer our research questions, we first recruited experienced practitioners
from both the arts and the cybersecurity field using a two-mode approach.
Then, we interviewed the participants to further understand their security and
privacy perceptions regarding immersive arts. From the interview transcripts,
we identified various themes as part of our analysis. To be eligible to
participate in the study, participants had to be either media artists or
cybersecurity experts and meet the following requirements: participants should
have five or more years of experience in their respective field; participants
should be U.S. residents; additionally for media artists, participants should
have a portfolio that includes interactive artwork installations, especially
installations that collected and/or displayed audience data.

Our recruitment took place via two modes: snowball sampling and a publicly
distributed screening survey. We used snowball sampling by emailing local
artists and cybersecurity experts with the required experience and asking if
they would be interested in participating in our study. We obtained their
contact information through their online portfolios. We asked those who were
interested to refer us to others in their field, who they knew would be
eligible to participate for this study. Additionally, we released a screening
survey on social media, inviting interested individuals with five or more
years of experience in either art or cybersecurity to fill it out. We
contacted respondents who successfully completed the survey and met our study
requirements to schedule interviews.

All participants who agreed to participate as a result of snowball sampling
and those who were contacted from the survey were asked to schedule time with
the research assistant for an interview. To better understand the
participants’ experiences in their respective fields before the interview, we
asked the participants for either a LinkedIn profile or online portfolio that
we could review in advance. The interviews were conducted remotely over Zoom
[18]. During the interview, we asked questions regarding the participants’
experiences in their respective fields, their personal experiences with
interactive art installations (either building them or interacting with them),
and what security and privacy concerns they had with the interactive art
installations they have built and/or experienced. Each interview was video
recorded, after obtaining participants’ permission through verbal consent.
Each interview lasted about an hour. After the interview, each participant was
awarded a $100 gift card for their time. The transcript for each video was
generated using the Temi transcription tool [14]. All videos and transcripts
were stored on the University of Denver OneDrive server.

To analyze the interviews, the transcripts were open-coded by the researchers
to recognize overall common themes among artists and cybersecurity experts.
These themes were divided into three main categories: (i) pain points or
concerns about the data collection by interactive art installations; (ii)
trends in how people educated themselves or taught others about cybersecurity
practices; and (iii) trends in software used to build secure technology.

##  3 Results

A total of five participants, comprising two media artist and three
cybersecurity experts, were interviewed. The participants’ backgrounds varied,
yet all had encountered interactive art installations in some capacity during
their professional careers. Notably, one participant had direct involvement in
building such installations, while another had contributed to research on
interactive technologies such as virtual reality (VR). The remaining
participants had professional ties to the field through collaboration with
colleagues or mentoring individuals involved in interactive media.

###  3.1 Immersive Art Data Collection Concerns

Regarding data collection within interactive installations, participants
identified several common types of data, including video, audio, voluntarily
supplied information such as survey responses, location data, and phone
numbers. In the context of interactive artworks, this data collection is
essential for understanding audience engagement and enhancing the
interactivity of the installations. However, while some data were voluntarily
provided by audience members, others, such as data provided by “proximity
sensors," were collected without explicit consent. Participants underscored
the importance of implementing opt-in systems to ensure transparency and user
consent. They highlighted the complexity of distinguishing between opt-in and
opt-out scenarios, acknowledging that current opt-in systems are often
intentionally difficult to understand and navigate. This complexity is
particularly problematic in interactive artworks, where the seamless and
immersive experience should not be compromised by convoluted consent
processes. Moreover, participants emphasized the need to collect only
necessary data and ensure compliance with relevant data protection regulations
such as the General Data Protection Regulation (GDPR) and the California
Consumer Privacy Act (CCPA).

Despite the inherent challenges in achieving complete security within
interactive installations, participants advocated for a balanced approach that
acknowledges the impossibility of eliminating all risks. This approach
involves proactive threat modeling, risk acceptance, and a duty of care to
protect audience members from potential harm arising from data misuse. They
stressed that interactive artworks, which often employ cutting-edge technology
and innovative user interactions, present unique cybersecurity challenges. One
of the participant emphasized:

> “There should be a ‘duty of care’ involved to protect [the audience] as much
> as possible… taking into account all the different ways that [the data]
> could be misused to harm them."

By integrating better cybersecurity measures tailored to the specific needs of
interactive artworks, creators can ensure that their installations not only
provide engaging and immersive experiences but also protect the privacy and
security of their audiences. Participants advocated for continuous education
and adaptation to evolving security requirements and best practices,
recognizing the dynamic nature of both technology and the threats it faces.

###  3.2 Continuous Education for Multiple Interdisciplinary Stakeholders

Participants also emphasized the importance of engaging space managers, such
as gallery employees, and implementing guidelines to secure audience data and
minimize employee touch-points with interactive installations. They suggested
training programs for gallery staff to ensure they understand the importance
of data security and are equipped to handle any issues that may arise. These
training programs could cover topics such as recognizing potential security
threats, responding to data breaches, and maintaining the integrity of the
installations. In addition, participants recommended the development of clear
policies and procedures for data handling and security, tailored to the
specific needs of interactive art installations. In terms of staying abreast
of evolving security requirements and practices, participants highlighted the
importance of adaptability and continuous learning. Collaborative spaces,
including workshops, lectures, and online communities, were cited as valuable
resources for exchanging knowledge and staying informed about emerging
security trends and solutions. These collaborative efforts can foster a
culture of continuous improvement and innovation in cybersecurity practices
within the interactive art community. Participants noted that these
collaborative spaces provide opportunities for cross-disciplinary learning,
where artists can gain insights from cybersecurity experts and vice versa.

Moreover, most cybersecurity experts expressed a keen interest in deepening
their understanding of data security issues and solutions specific to
interactive art installations. Many admitted to previously overlooking or
underestimating the data security implications of the installations they
encountered, indicating a growing awareness and interest in this domain. This
increased awareness was often spurred by recent high-profile data breaches and
privacy scandals, which underscored the importance of integrating security
measures into all technological implementations, including art installations.
Participants recognized that the evolving nature of cybersecurity threats
requires a dynamic and responsive approach, with continuous education and
adaptation being key components.

###  3.3 Secure and Privacy-focused Software for Immersive Art

A recurring theme among participants was the concern for better cybersecurity
measures tailored to the specific needs of interactive artworks. Participants
emphasized the need for software that is accessible and aligns with the
project’s contextual requirements while ensuring an appropriate level of
security and privacy. One participant highlighted their preference for using
Signal, an end-to-end encrypted communication platform, citing its ease of use
and adaptability to various technological contexts. In the context of
interactive artworks, they stressed the importance of secure communication
among collaborators to protect sensitive creative discussions and proprietary
project details for which messaging apps such as Signal can be used. They
emphasized the significance of considering factors such as file size and
internet bandwidth when selecting communication platforms to ensure
accessibility for all collaborators when it comes to artwork discussions. The
participant articulated their approach, stating:

> “How difficult is it for them in the region that they’re in to access the
> [software] that I’m recommending? … If it’s a really large file to download,
> will it download across 3G or 4G? Will it work on the kind of phone [other
> collaborators] have?"

##  4 Discussion

The findings of this study illuminate the nuanced approaches adopted by both
cybersecurity experts and media artists in addressing data security concerns
within the realm of interactive art installations. Through a comprehensive
analysis of the data, several key themes and insights emerge, shedding light
on the intricate interplay between technology, creativity, and security. This
study underscores the dynamic nature of cybersecurity and artistic practices,
characterized by continual adaptation and innovation. Both cohorts exhibit a
proactive stance towards technological security, emphasizing the importance of
minimizing data collection, ensuring secure storage, and effectively
communicating security protocols with participating audiences. This
multifaceted approach reflects a shared commitment to protecting privacy while
facilitating engaging interactive experiences.

One prominent theme is the emphasis on the importance of legal frameworks and
compliance standards. The influence of regulations such as the GDPR or CCPA is
evident in the practices of both cybersecurity experts and media artists.
These legal frameworks provide a strategic imperative for assessing and
mitigating risks associated with data storage and handling [1]. While
cybersecurity experts caution against the illusion of absolute security, they
advocate for a pragmatic approach to risk management and risk acceptance. This
approach acknowledges the impossibility of eliminating all risks while
striving to minimize potential vulnerabilities. A consensus among study
participants revolves around the adoption of protective data handling
strategies, particularly advocating for a more transparent and user-centric
approach. The preference for an opt-in model for data interpretation and
storage underscores a commitment to user privacy and autonomy [15, 10]. By
prioritizing minimalistic data collection and empowering interaction
participants to make informed choices regarding their data, artists may seek
to foster a culture of responsible data handling within interactive art
contexts. This practice not only promotes trust and transparency in the
digital realm but also aligns with ethical considerations in data security and
honors privacy regulations that impact the users.

Another significant insight is the role of experiential learning in enhancing
understanding and awareness of data security practices. Participants express a
preference for hands-on experiences, citing interactive art installations as
potent educational tools for demystifying concepts of data storage and
security. These installations serve as immersive environments where audiences
can engage with security concepts in a tangible and intuitive manner. Such
initiatives not only enrich the artistic experience but also contribute to
broader conversations surrounding cybersecurity best practices. This
underscores the transformative potential of art as a vehicle for knowledge
dissemination and social change. The study also highlights the importance of
continuous education and adaptation to evolving security requirements. Both
cybersecurity experts and media artists recognize the dynamic nature of
technology and the associated security threats. Collaborative spaces,
including workshops, lectures, and online communities, are cited as valuable
resources for exchanging knowledge and staying informed about emerging
security trends and solutions. These collaborative efforts foster a culture of
continuous improvement and innovation in cybersecurity practices within the
interactive art community. Participants noted that these collaborative spaces
provide opportunities for cross-disciplinary learning, where artists can gain
insights from cybersecurity experts and vice versa.

Moreover, the study reveals a growing awareness among cybersecurity experts of
the specific data security issues and solutions related to interactive art
installations. Many experts admitted to previously overlooking or
underestimating the data security implications of the installations they
encountered. This increased awareness is often spurred by recent high-profile
data breaches and privacy scandals, which underscore the importance of
integrating security measures into all technological implementations,
including art installations. The evolving nature of cybersecurity threats
requires a dynamic and responsive approach, with continuous education and
adaptation being key components.

##  5 Conclusion

The synergy between interactive artworks and cybersecurity is essential for
creating safe and engaging experiences for audiences. By implementing enhanced
security measures, transparent consent systems, and ongoing education and
collaboration, the interactive art community can effectively address the
unique challenges posed by data collection and privacy concerns. Our work with
five professionals in cybersecurity and art highlights the necessity for a
proactive and balanced approach to ensure that the creative and experiential
aspects of interactive art are preserved while enhancing the privacy and
security of audience members. By examining the perspectives and practices of
cybersecurity experts and media artists, this study offers a nuanced
understanding of the challenges and opportunities inherent in reconciling
artistic expression with data security imperatives. This understanding paves
the way for informed discourse and collaborative action in protecting privacy
and promoting responsible innovation in the ever-evolving digital age. The
insights gained from this study emphasize the importance of continuous
adaptation and learning, fostering a culture of responsible data handling and
ethical practices within the interactive art community. Through these efforts,
the interactive art space can continue to thrive as a dynamic and secure
environment, benefiting both creators and audiences alike.

##  6 Acknowledgement

This work was supported by the CAHSS Dean’s Award for Interdisciplinary
Studies (DAIS) and the Faculty Research Fund for Research for the Privacy and
Security of Digital Arts. We would also like to thank our participants for
this study and acknowledge the Inclusive Security and Privacy-focused
Innovative Research in Information Technology (InSPIRIT) Lab at the University
of Denver for supporting this work. Any opinions, findings, conclusions, or
recommendations expressed in this material are solely those of the authors.

## References

  * [1] Andrick Adhikari, Sanchari Das, and Rinku Dewri.  Evolution of composition, readability, and structure of privacy policies over two decades.  Proceedings on Privacy Enhancing Technologies, 2023.
  * [2] Sanchari Das and Kate Hollenbach.  A proposal for understanding professional needs to create privacy-preserving and secure emergent digital artworks.  In Proceedings of the 7th Workshop on Inclusive Privacy and Security (WIPS) in Association with the USENIX Symposium on Usable Privacy and Security (SOUPS), 2022.
  * [3] Ioannis Deliyannis.  Interactive multimedia installation art development using recycled input and sensing devices.  International Journal of Arts and Technology, 9(2):108–125, 2016.
  * [4] Emily Grenader, Danilo Gasques Rodrigues, Fernando Nos, and Nadir Weibel.  The videomob interactive art installation connecting strangers through inclusive digital crowds.  ACM Transactions on Interactive Intelligent Systems (TiiS), 5(2):1–31, 2015.
  * [5] Design I/O.  Design I/O - Interactive Installations — design-io.com.  https://www.design-io.com/.  [Accessed 23-05-2024].
  * [6] Golan Levin.  Painterly interfaces for audiovisual performance.  PhD thesis, Massachusetts Institute of Technology, 2000.
  * [7] Golan Levin and Zachary Lieberman.  In-situ speech visualization in real-time interactive installation and performance.  In NPAR, volume 4, pages 7–14. Citeseer, 2004.
  * [8] Rafael Lozano-Hemmer.  Rafael lozano-hemmer.  Radio Program. Broadcast June, 18:10, 2010.
  * [9] Sasha Lukyanova.  Look but also touch: the rise of immersive art | Basa Studio — basa-studio.com.  https://basa-studio.com/stories/the-rise-of-immersive-art.  [Accessed 23-05-2024].
  * [10] Philipp Markert, Andrick Adhikari, and Sanchari Das.  A transcontinental analysis of account remediation protocols of popular websites.  arXiv preprint arXiv:2302.01401, 2023.
  * [11] Lauren McCarthy, Casey Reas, and Ben Fry.  Getting started with P5. js: Making interactive graphics in JavaScript and processing.  Maker Media, Inc., 2015.
  * [12] Teague McDaniel.  Teague mcdaniel - cargo site.  https://teaguemcdaniel.cargo.site/.  Accessed: 2024-05-23.
  * [13] Kyle McDonald.  Only everything lasts forever.  PhD thesis, Rensselaer Polytechnic Institute, Troy, NY, 2010.
  * [14] Temi.  Temi - audio and video transcription service.  https://www.temi.com/.  Accessed: 2024-05-23.
  * [15] Kathryn Walsh, Faiza Tazi, Philipp Markert, and Sanchari Das.  My account is compromised-what do i do? towards an intercultural analysis of account remediation for websites.  In Proceedings of the Sixth Workshop on Inclusive Privacy and Security (WIPS 2021): In Association with the Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021), 2021.
  * [16] Miri Weiss Cohen, Alex Frid, Mark Malin, and Vladimir Ladijenski.  Generating 3d cad art from human gestures using kinect depth sensor.  Computer-Aided Design and Applications, 12(5):608–616, 2015.
  * [17] Simon Wu.  Sam lavigne and tega brain: New york apartment.  Brooklyn Rail, 2020.
  * [18] Inc. Zoom Video Communications.  Zoom video conferencing.  https://zoom.us/.  Accessed: 2024-05-23.


# ConTIG: Continuous Representation Learning on Temporal Interaction Graphs

Xu Yan, Xiaoliang Fan, Peizhen Yang, Zonghan Wu, Shirui Pan, Longbiao Chen, Yu
Zang, and Cheng Wang  Xu Yan, Xiaoliang Fan (the corresponding author),
Peizhen Yang, Longbiao Chen, Yu Zang and Cheng Wang are with Fujian Key
Laboratory of Sensing and Computing for Smart Cities, School of Informatics,
Computer Science & Technology Department, Xiamen University, Xiamen, China.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Zonghan Wu is with Centre for Artificial Intelligence, FEIT,
University of Technology Sydney, Australia.
E-mail<EMAIL_ADDRESS>Shirui Pan is with Department of Data
Science and AI, Faculty of Information Technology, Monash University,
Australia.
E-mail<EMAIL_ADDRESS>
###### Abstract

Representation learning on temporal interaction graphs (TIG) is to model
complex networks with the dynamic evolution of interactions arising in a broad
spectrum of problems. Existing dynamic embedding methods on TIG discretely
update node embeddings merely when an interaction occurs. They fail to capture
the continuous dynamic evolution of embedding trajectories of nodes. In this
paper, we propose a two-module framework named ConTIG, a continuous
representation method that captures the continuous dynamic evolution of node
embedding trajectories. With two essential modules, our model exploit three-
fold factors in dynamic networks which include latest interaction, neighbor
features and inherent characteristics. In the first update module, we employ a
continuous inference block to learn the nodes’ state trajectories by learning
from time-adjacent interaction patterns between node pairs using ordinary
differential equations. In the second transform module, we introduce a self-
attention mechanism to predict future node embeddings by aggregating
historical temporal interaction information. Experiments results demonstrate
the superiority of ConTIG on temporal link prediction, temporal node
recommendation and dynamic node classification tasks compared with a range of
state-of-the-art baselines, especially for long-interval interactions
prediction.

###### Index Terms:

Graph Embedding, Temporal Interaction Graph, Neural Ordinary Differential
Equations, Graph Neural Networks.

##  1 Introduction

Graph representation learning has attracted a surge of research attention
owing to the widespread existence of graph-structured data in the real world
such as social networks, citation networks, and other user-item interaction
systems [1]. Learning graph embedding is a powerful approach of graph
representation learning, which maps the characteristics of nodes to a low-
dimensional vector space so that the proximities of nodes in topological space
can be well reserved [2]. It has shown a great success in graph representation
learning from shallow graph embedding methods [3, 4, 5] to deep graph neural
networks (GNNs) [6, 7, 8, 9].

Recently, representation learning on dynamic graph has attracted many research
attention [10, 11, 12], and they mainly model temporal graphs either as a
sequence of snapshots [13, 14, 15, 16, 17, 18], or as real events with
timestamps [19, 20, 21, 22, 23, 24].

For existing works learning temporal interaction embedding from the sequence
of interactions, a number of studies discretely update the embeddings of the
interactive nodes once an interaction occurs [25, 26]. Other works learn the
interactions by aggregating the temporal neighbor features to pass messages
between nodes [19, 23], which model discrete dynamic of node representation in
multiple propagation layers [27]. Although these discrete works have made
substantial advances in learning temporal interaction embedding, they fail to
capture the continuous dynamic evolution of nodes embedding trajectories thus
losing temporal information for non-ignorable and inactive nodes with long-
interval interactions (e.g., User A in Fig.1). Therefore, we aim to learn
nodes embedding trajectories and capture the continuous dynamic of node
representations.

Figure 1: An illustrated example of the inactive user (i.e., User A) with
long-interval interaction in a temporal interaction graph. In the long-
interval of User A, her future posts about COVID-19 might be affected by three
factors: 1) her latest interaction with “Gathering guidelines”; 2) her
neighbors, user B and C; and 3) her inherent characteristics as a healthcare
personnel.

Learning dynamic node embedding trajectories is extremely challenging due to
the complex non-linear dynamic in temporal interaction graphs (TIG). We
conclude the challenges in three folds under the scenario of posting in
Reddit111https://www.reddit.com/ website. First, latest interaction
information may have a significant impact on the current interaction, e.g., in
Fig.1, the posts made by User A at $t_{4}$ would be influential for $t_{q}$,
since guidelines of holiday gathering under COVID-19 pandemics will be
regularly implemented. Second, the node state is also affected by their
neighbors’ attention over time. For instance, User A is expected to emulate
their neighbors by posting “COVID-19 vaccine” at $t_{q}$ in Fig.1. Third,
inherent characteristics of node would fatally determine the state regardless
of aforementioned two factors. For example, User A is a frontline healthcare
personnel and she could inherently pay much attention to COVID-19 trainings on
infection control. In summary, existing methods hold partial considerations
for these three-fold factors.

To cover the shortcomings of previous methods, we propose a Continuous
representation learning method on Temporal Interaction Graphs (ConTIG). The
proposed ConTIG contains two modules: the update module and transform module.
When an interactive message comes, in the update module, inspired by [28], we
define a neural ordinary differential equations (ODE)[29] with the three
aforementioned factors affecting node states (i.e., latest interaction,
neighbors’ feature and inherent characteristics), and incorporate them with
the continuous-time encoding function to trace dynamic knowledge and learn
node state trajectories. Then the node embeddings at the ending time of neural
ODE is used to update embeddings of interacting nodes. In the transform
module, a self-attention mechanism is introduced to awaken historical
interactive information of current interacting nodes and convert them to
generate future node representations. The results on temporal link prediction,
temporal node recommendation and dynamic node classification tasks show the
effectiveness of our proposed model compared with a range of state-of-the-art
baselines, especially for long-interval interactions prediction.

The contributions of this work are summarized as follows:

  * •

We propose a representation learning method to incorporate three-fold factors
(i.e., latest interaction, neighbor features and inherent characteristics)
with a neural ODE. This allows us to capture continuous dynamic of node
embedding trajectories by continuously updating node embeddings in the update
module.

  * •

We combine aforementioned update module with a self-attention mechanism to
predict future embeddings in the transform module. This makes ConTIG effective
on long-interval interactions prediction.

  * •

We evaluate our ConTIG on temporal link prediction, temporal node
recommendation and dynamic node classification tasks, and ConTIG compares
favorably against state-of-the-art methods.

The rest of this paper is organized as follows. In Section 2, we discuss some
related work. Sections 3 and 4 describes the notations and our proposed model
in detail. In Section 5, we conduct experiments on several datasets and
compare them with state-of-the-art methods. In Section 6, the conclusion and
our future work are presented.

##  2 Related Work

###  2.1 Static Graph Embedding

Early works for representation learning are mainly shallow models including
graph factorization approaches [30, 31] and skip-gram models [3, 4, 5], which
learn node representations by random walk objectives. With the success of deep
learning, GNNs [8, 7, 9, 32] have gradually attracted great research interest.
They are effective approaches to learn node representations by updating each
node according to messages from neighboring nodes in graphs in each layer.
GNNs essentially model discrete dynamics of node representations with multiple
propagation layers [27]. However, all the above mentioned approaches are
limited to learning node embeddings on static graph structure information, and
the temporal behavior of interaction over time is generally ignored.

###  2.2 Temporal Graph Embedding

One common way to handle temporal graphs is to decompose it into multiple
static graphs snapshots by a regular time interval. Some works embed the graph
convolution into the recurrent neural network (RNN) based models or attention
mechanism [33], which learns to exploit the dynamic information in the graph
evolution within a period of time [13, 34, 35, 14, 16, 17, 36]. Some other
works are dynamic extensions of ideas applied in the static case inspired by
methods such as PageRank[37], graph autoencoder[38] and the topic model [15]
to capture both the temporal community dynamics and evolution of graph
structure. However, learning embedding from the sequence of graph snapshots
sampled from the temporal graph by a regular time interval may lose
information by looking only at some snapshots of the graph over time.

Therefore, recent works learn temporal graph embedding from the sequence of
timed interactions. A number of studies learn the sequence of interactions by
discretely updating the node embeddings once an interaction occurs by RNNs
[25, 39, 24], memory networks [26], time point process [40, 41], transformer
network[42], generative models [43], etc. Other works learn the interactions
by aggregating the temporal neighbor features to pass messages between nodes
[20, 23, 44, 24] or learning node representation from random walk objects [21]
and temporal motifs [45, 46, 47]. TGAT [23] proposes the temporal graph
attention layer to aggregate temporal-topological neighborhood features. TGN
[24] makes a combination of updating operators and aggregating operators.
However, these methods learn the discrete dynamic of node representations, and
it is not beneficial to learn the node state trajectories. Moreover, they
ignore the influence of the inherent characteristics of the change on the node
states, which is not beneficial to capture the complex non-linear dynamic of
node representations. Different from the aforementioned works, we assembled
the latest interaction, neighbor features and inherent characteristics of the
nodes into a neural ODE to learn node state trajectories and update node
embeddings. As a result, our method captures the continuous dynamic of node
representation.

###  2.3 Neural ODE

Neural ordinary differential equations (ODE)[29] is an approach for modelling
a continuous dynamics on hidden representation, where the dynamic is
characterised through an ODE parameterised by a neural network (i.e. ODE
Solver). Recently, there are some works devote to use neural ODE [29] combined
with GNN to characterise the continuous dynamics of node representations [28,
48]. However, these methods are not built for temporal graphs, which are
continuous-time GNNs learning the dynamic of node representation in static
graph scenario to build ”deeper” networks. For temporal graphs, [49] learns
the evolution of the temporal knowledge graph dynamics over time using ODEs as
a tool. CG-ODE [50] proposed a latent ODE generative model that learns the
coupled dynamics of nodes and edges with a GNN based ODE in a continuous
manner. Different from them (i.e., [49, 50]), we stand at the perspective of
node state modeling in temporal interaction graph, using a differential
equation integrating the three-fold factors (i.e., latest interaction,
neighbor features and inherent characteristics). This allows us to
successfully capture continuous dynamic of node embedding trajectories.

##  3 Problem Definition

We summarize some notations and state the problem we want to solve in this
section.

######  Definition 1 (Temporal interaction graph)

Temporal interaction graph is a graph with temporal and attributed interaction
edges between nodes. Temporal interaction graph is defined as a pair
$\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ represents
vertices sets, and $\mathcal{E}$ is a set of the sequences of interactions
with time label between two vertices. An interaction $e$ is a tuple of form
$(u,i,t,f^{ui})$, where $u$ and $i\in\mathcal{V}$ represent two vertices have
an interaction at timestamp $t$, and $f^{ui}$ represents the feature of
interaction $e$.

######  Definition 2 (Temporal interaction graph embedding)

Given the temporal interaction graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$,
we aim to learn a mapping function:
$\phi:\mathcal{V}\rightarrow\mathbb{R}^{d}$ to map node features into a low-
dimensional latent space and generate node embeddings
$H\in\mathbb{R}^{\left|\mathcal{V}\right|\times d}$ to represent nodes, where
$d\ll\left|\mathcal{V}\right|$ representing the dimension of node embeddings.

######  Definition 3 (Interaction intervals)

In the temporal interaction graph, each interaction is an edge connected by a
source node and a target node, so we can use source nodes as the label of
interaction to calculate the time interval $\Delta t_{e}$ between current
interaction and its latest interaction, where the latest interaction is the
edge where the source node last appeared. For example, the latest interaction
of the current interaction, $e_{p}^{j}=(u^{j},i,t_{p},f^{ui})$, is the
interaction $e_{q}^{j-1}=(u^{j-1},r_{u},\tau_{u},f^{ur_{u}})$ in which the
node $u$ at the last time appeared. As a result, the time interval $\Delta
t_{e_{p}}$ between $e_{p}^{j}$ and $e_{q}^{j-1}$ is $t_{p}-\tau_{u}$, where
$r_{u}$ and $\tau_{u}$ represents the latest interactive node and timestamp of
$u$, respectively. While, $p$ and $q$ represent the interaction id, and $j$
represents that how many times $u$ appears. Here, if the node appears for the
first time (i.e., $j=0$), the time interval of the interaction is $0$.

##  4 The Proposed Model

In this section, we will describe our model in detail. The proposed model –
ConTIG consists of two essential modules (i.e., update and transform module).
We will briefly introduce them in Section 4.1 and describe each module in
detail in the rest of sections.

###  4.1 Overview of the ConTIG

Figure 2: ConTIG Framework. The encoder-decoder framework (left) consists of a
continuous inference block (middle) and a graph attention layer (right). The
former contains a neural ODE defined with three-fold factors (i.e., latest
interaction, neighbors’ feature and inherent characteristics) to update the
node embeddings according to the output of encoder. The latter estimates the
future embedding of nodes by aggregating observed historical temporal
interaction information.

Fig.2 provides the framework of our proposed ConTIG. The encoder-decoder
framework (Fig.2 left) consists of two modules: the update module and the
transform module. In the update module (Fig.2 middle), a continuous inference
block is used to update the embeddings of the interacting nodes in a
continuous setting. In the transform module (Fig.2 right), a graph attention
layer is applied to estimate the future embedding of nodes by capturing
observed historical temporal interaction information. When an interaction
message $e_{p}=(u,i,t_{p},f^{ui})$ comes, we first employ a neural encoder to
project the latest interaction message $e_{q}=(u,r_{u},\tau_{u},f^{ur_{u}})$
of input interacting node into a latent space, $i.e.,E=f(X)$. Afterwards,
treating $E$ as the initial value $H^{t_{p}}\left(0\right)$ in continuous
inference block, we utilize a neural ordinary differential equations (ODE) to
learn the nodes’ continuously changing state trajectories at a certain time
interval $\left[0,\Delta t\right]$, and the embeddings of nodes
$\bar{H}^{t_{p-1}}_{u}$ is updated as $\bar{H}^{t_{p}}_{u}$ by the node
embeddings obtained in nodes’ state trajectory at ending time
$H^{t_{p}}\left(\Delta t\right)$. Then, selecting $k$ observed historical
neighbors of current nodes $u$, we introduce a self-attention mechanism on
graphs to convert these interactions information to generate future node
embedding $\hat{H}^{t}_{u}$. Finally, the decoder uses the future node
embeddings $\hat{H}^{t}_{u}$ for downstream tasks (i.e., temporal link
prediction, temporal node recommendation and dynamic node classification).

###  4.2 Encoder-Decoder

The proposed ConTIG adapts an encoder-decoder architecture (Fig.2 left). For
each interaction, we first project the features into a latent space to
generate the initial hidden representation of nodes. After learning node
embeddings at current timestamp, we finally design a decoder for the specific
downstream tasks.

Before the encoder, we initialize the latest interactive node $r_{v}$ and
timestamp $\tau_{v}$ as $0$ for all node $v\in\mathrm{V}$ (line 1 in Algorithm
1).

In the encoder, to learn the temporal pattern of each interaction, we use a
time encoding function in [51] to obtain a continuous functional mapping
$F_{T}:T\rightarrow\mathbb{R}^{d_{T}}$ from time domain to the
$d^{T}$-dimensional vector space, which projects the timestamps of
interactions into the continuous-time space. For any given time $t$, we will
generate its time embeddings as follows:

$\begin{split}F_{T}(t)=\frac{1}{\sqrt{d_{T}}}[cos\left(\omega_{1}t\right)&,sin\left(\omega_{1}t\right),\\\
...&,cos\left(\omega_{d_{T}}t\right),sin\left(\omega_{d_{T}}t\right)],\end{split}$
(1)

where $\omega_{1},\omega_{2},...,\omega_{d^{T}}$ are trainable parameters. To
learn informative node representations $H^{t_{p}}$ at timestamp $t_{p}$, we
concatenate the latest node embedding of the interactive node
$\bar{H}^{t_{p-1}}_{u}$ and it latest interactive node
$\bar{H}^{t_{p-1}}_{r_{u}}$, the last interaction message feature $f^{ur_{u}}$
and the time embeddings of the time interval $F_{T}(\Delta t)$ to represent
new node features of $u$ at timestamp $t$ (line 3 in Algorithm 1). Here, as we
focus on the interval between current interaction and its latest interaction,
we use the time embedding $F_{T}(\Delta t)$. Afterwards, we adopt a fully
connected layer as an encoder (line 4 in Algorithm 1), and the hidden
representations can be defined as follows:

$\displaystyle E$
$\displaystyle=f(\bar{H}^{t_{p-1}}_{u}\left|\right|\bar{H}^{t_{p-1}}_{r_{u}}\left|\right|f^{ur_{u}}\left|\right|F_{T}(\Delta
t)),$ (2)

where $\left|\right|$ is the concatenation operation, and $f(\cdot)$ is a
linear projection as follows:

$\displaystyle f(x)=xW+b,$ (3)

where $W$ and $b$ are learnable parameters.

For the decoder, two fully connected layers are designed for temporal link
prediction and temporal node recommendation tasks, and three fully connected
layers are designed for dynamic node classification task.

###  4.3 Update: Continuous Inference Block

In update module (Fig.2 middle), to capture the continuous dynamic of node
representation, inspired by [28], we define a ODE with the latest interaction
information, neighbor features and inherent characteristics of nodes, and use
a neural solver to generate the node state trajectories at a certain time
interval $\left[0,\Delta t\right]$. In this way, our method can estimate the
possible direction of embedding trajectory for an inactive node (line 5 and
11-16 in Algorithm 1.

We select the interactions between each node in temporal interaction graph $u$
and its last interactive node $r_{u}$, and divide them into a set
$\mathcal{E}_{p}$ to generate an adjacency matrix $S_{p}$ describing their
relationships.
$S_{p}\in\mathbb{R}^{\left|\mathcal{V}\right|\times\left|\mathcal{V}\right|}$
is defined by $\mathcal{E}_{p}$ as follows:

$\displaystyle S_{p}^{ui}$ $\displaystyle=\begin{cases}1&\text{ if
}(u,r_{u})\in\mathcal{E}_{p}\\\ 0&\text{ otherwise},\end{cases}$ (4)

As the degree of nodes can be very different, we typically normalize the
adjacency matrix as $D_{p}^{-\frac{1}{2}}S_{p}D_{p}^{-\frac{1}{2}}$, where
$D_{p}=\mathrm{diag}\left(\sum_{j}S_{p}^{ij}\right)\in\mathbb{R}^{M\times M}$
is the degree matrix of $S_{p}$. As such a normalized adjacency matrix always
has an eigenvalue decomposition, and the eigenvalues are in the interval
$\left[-1,1\right]$. To get the positive eigenvalues and make graph learning
algorithms stable in practice, we follow [7] and leverage the following
regularized matrix for characterizing graph structures:

$\displaystyle A_{p}$
$\displaystyle=\frac{\beta}{2}\left(I_{N}+D_{p}^{-\frac{1}{2}}S_{p}D_{p}^{-\frac{1}{2}}\right),$
(5)

where $\beta\in\left(0,1\right)$ is a hyperparameter,
$I_{N}\in\mathbb{R}^{N\times N}$ is the identity matrix,
$N=\left|\mathrm{V}\right|$, and eigenvalues of $A_{p}$ are in
$\left[0,\beta\right]$.

Afterwards, to capture the complex non-linear dynamic of node embeddings, we
assume that there are three possible factors affecting the node states: 1)
latest interaction information exhibiting the latest nodes states; 2)
neighbors’ features affecting the change of node states; 3) the inherent
characteristics of nodes determining the influence of the aforementioned two
factors. Based on these considerations, we define an ODE in $\mathcal{E}_{p}$
to solve the node embedding trajectories, which adaptively fuse them with a
gated mechanism. Here, we treat the encoder $E_{t_{p}}$ as the initial value
of ODE $H^{t_{p}}(0)$, i.e., $H^{t_{p}}(0)=E_{t_{p}}$. Then we use a
differential equation defined as follows to learn the continuous node
representations in the interval of interactions:

$\displaystyle\frac{\mathrm{d}H^{t_{p}}(t)}{\mathrm{d}t}=\underset{\text{latest
interaction}}{\underbrace{z_{l}\odot H^{t_{p}}(0)}}+\underset{\text{neighbor
features}}{\underbrace{z_{n}\odot A_{p}H^{t_{p}}(t)}}-\underset{\text{inherent
characteristics}}{\underbrace{z_{i}\odot H^{t_{p}}(t)}},$ (6)

where $H^{t_{p}}(0)$ represents the latest interaction information of nodes,
$A_{p}H^{t_{p}}(t)$ represents the influence from neighbors, $-H^{t_{p}}(t)$
represents the inherent characteristics of nodes, $\odot$ denotes the element-
wise product operation. $z_{l}$, $z_{n}$, and $z_{i}$ represents three gates,
respectively. They are computed as:

$\displaystyle z_{l}$ $\displaystyle=\sigma(W_{l}H^{t_{p}}(0)+b_{l}),$ (7)
$\displaystyle z_{n}$ $\displaystyle=\sigma(W_{n}H^{t_{p}}(0)+b_{n}),$ (8)
$\displaystyle z_{i}$ $\displaystyle=\sigma(W_{i}H^{t_{p}}(0)+b_{i}),$ (9)

where $W_{l},W_{n},W_{i}$ and $b_{l},b_{n},b_{i}$ are trainable parameters,
and $\sigma(\cdot)$ is the sigmoid activation function to normalize the output
into $[0,1]$. The gated fusion mechanism can adaptively fuse three factors
according to their importance calculated by the initial value of ODE. In
addition, the updating process starts at the timestamp of latest interaction
of the node. To this end, following [24], when an interactive message comes,
we use the latest interaction information to update node embeddings and save
the current interaction message for the next time the node appears. Meanwhile,
due to the integration of time encoding in $H^{t_{p}}(0)$, the temporal
behaviors of nodes could be captured. As a result, we can leverage the
aforementioned ODE to learn the nodes’ state trajectories at the interval
$\left[0,\Delta t\right]$, and the three main factors mentioned above jointly
captures the continuous dynamic of node representations. Then the node
representations are updated by ODE solver as follows:

$\displaystyle H^{t_{p}}(\Delta t)=\mathrm{ODESolve}(g(t),H^{t_{p}}(0),\Delta
t)$ (10)

where $g(t)=\frac{\mathrm{d}H^{t_{p}}(t)}{\mathrm{d}t}$.

Finally, we use the hidden state at the end time $H(\Delta t_{p})$ to update
the previous embedding memory $\bar{H}^{t_{p-1}}$ as $\bar{H}^{t_{p}}$ (i.e.,
$\bar{H}^{t_{p}}=H(\Delta t_{p})$) (line 5 in Algorithm 1).

###  4.4 Transform: Graph Attention Layer

After capturing the continuous dynamic of node representations from latest
interactions sets and updating the embeddings of nodes, a graph attention
layer in transform module (Fig.2 right) is applied to convert the historical
observed interaction features of nodes to generate future representations
(Line 6 and 17-24 in Algorithm 1).

In this module, for the current interaction $e$ connected by $u$ and $v$ at
time $t$, their temporal neighbors and the interaction information between
them are token as input. We introduce a self-attention mechanism to
distinguish different neighbors, and take account of the structural
information with temporal information [23]. For node $u$ at time $t$, we
consider its neighbors $N(u,t)=\\{i_{0},...,i_{k-1}\\}$, where the
interactions between $u$ and $i_{j}\in N(u,t)$ occurred at time $t_{j}$ prior
to time $t$, and the sampling process of temporal neighbors of node $i$ is the
same as $u$. Then we take the node information with time $t$ encoding
$Z^{t}_{u}=\bar{h}^{t}_{u}\left|\right|F_{T}(t)$ and the neighborhood
information with the time delta $t-t_{j}$ encoding
$Z^{t}_{N}=\bar{H}^{t}_{N(u,t)}\left|\right|f^{uN(u,t)}\left|\right|F_{T}(t-t_{j})$
as the input of attention, where the time delta $t-t_{j}$ is between current
interaction $e$ and the interaction of $u$ and its neighbors $i_{j}\in
N(u,t)$. In attention, three different linear projections is used to obtain
the query $Q_{t}$, key $K_{t}$ and value $V_{t}$:

$\displaystyle
q_{t}=Z^{t}_{u}W_{Q},K_{t}=Z^{t}_{N}W_{K},V_{t}=Z^{t}_{N}W_{V},$ (11)

where $W_{Q},W_{K},W_{V}$ are the weight matrices. The attention weights
$\alpha_{j}$ is given by:

$\displaystyle\alpha_{j}=\frac{\exp(q^{\mathrm{T}}K_{j}/\sqrt{d})}{\textstyle\sum_{c}\exp\left(q^{\mathrm{T}}K_{c}/\sqrt{d}\right)},$
(12)

where the attention weight $\alpha_{j}$ reveals how node $u$ attends to the
features of node $i_{j}$ within the topological structure defined as $N(u,t)$
after accounting for their interaction time with $i_{j}$. Therefore, the layer
produces the time-aware representation of node $u$ at time $t$,
$\hat{h}_{u}^{t}$, which represents the future hidden representations
generated by historical observed interactions as follows:

$\displaystyle\hat{h}_{u}^{t}$
$\displaystyle=\sum_{j=0}^{k-1}\alpha_{j}V_{t},$ (13)

To stabilize the learning process and improves performances, we extend the
self-attention mechanism to be multi-head ones [33]. Specifically, we
concatenate $M$ parallel attention mechanisms with different learnable
projections:

$\displaystyle{\hat{h}_{u}^{t}}$
$\displaystyle=\left|\right|_{m=1}^{M}\left\\{\sum_{j=0}^{k-1}\alpha^{(m)}_{j}V^{(m)}_{t}\right\\},$
(14)

where $V^{(m)}_{t}$ represent the value with different projections in the
$m^{th}$ head attention, and $\alpha^{(m)}_{j}$ represent the attention weight
calculated by the query and key with different projections in the $m^{th}$
head attention.

Finally, the future node embeddings $\hat{H}_{u}^{t_{p}}$ is generated by
calculating $\hat{h}_{u}^{t_{p}}$ for each node in interactions (line 6 in
Algorithm 1). The latest interactive node $r_{u}$ and $r_{i}$ are updated as
$i$ and $u$, respectively. In addition, the latest interactive timestamps
$\tau_{u}$ and $\tau_{i}$ are both updated as $t_{p}$ (Line 7-8 Algorithm 1).

###  4.5 Binary Cross-Entropy Loss Function

In this work, we adopt time-sensitive link prediction binary cross-entropy
loss function to learn ConTIG’s parameters. The binary cross-entropy loss
function is defined as follows and our goal is to maximize this likelihood
function:

$\begin{split}\mathcal{L}=\sum_{(u_{p},i_{p},t_{p})\in\mathcal{E}}&-\log\sigma({-\hat{h}_{u_{p}}^{t_{p}}}^{\mathrm{T}}\hat{h}_{i_{p}}^{t_{p}})\\\
&-Q\mathbb{E}_{i_{q}\sim
P(i)}\log{\sigma({-\hat{h}_{u_{p}}^{t_{p}}}^{\mathrm{T}}\hat{h}_{i_{q}}^{t_{p}}}),\end{split}$
(15)

where the summation is over the observed edges on $u_{p}$ and $i_{p}$ that
interact at time $t_{p}$, $Q$ is the number of negative samples and $P(i)$ is
the negative sampling distribution over the node space.

Algorithm 1 Continuous Representation Learning on Temporal Interaction Graphs

Input: Edge Stream $e=(u,i,t,f^{ui})\in\mathcal{E}$ in temporal interaction
graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$
Output: Embedding results ${\hat{h}_{u}^{t}}$ for all nodes

1:$\bar{h}_{v}^{0}\leftarrow x_{v},r_{v}\leftarrow 0,\tau_{v}\leftarrow
0,\forall v\in\mathcal{V}$

2:for $e_{p}=(u,i,t_{p},f^{ui})$ in $\mathcal{E}$ do

3: Learning $F_{T}(t_{p}-\tau_{v})$ by Eqn.1

4:
$E_{v}^{t_{p}}=f(\bar{h}_{v}^{t_{p-1}}\left|\right|\bar{h}_{r_{v}}^{t_{p-1}}\left|\right|f^{vr_{v}}\left|\right|F_{T}(t_{p}-\tau_{v})),\forall
v\in\mathcal{V}$

5: $\bar{h}^{t_{p}}_{u},\bar{h}^{t_{p}}_{i}\leftarrow$ Continuous Inference
Block $(E^{t_{p}},u,i)$

6: $\hat{h}^{t_{p}}_{u},\hat{h}^{t_{p}}_{i}\leftarrow$ Graph Attention Layer
$(\bar{h}^{t_{p}}_{u},\bar{h}^{t_{p}}_{i},u,i,t_{p})$

7: $r_{u}\leftarrow i,\tau_{u}\leftarrow t_{p}$

8: $r_{i}\leftarrow u,\tau_{i}\leftarrow t_{p}$

9:end for

10:return $\bar{h}_{v}^{t},\forall v\in\mathcal{V}$

11:procedure Continuous Inference Block($E^{t_{p}},u,i$)

12: $h_{v}(0)\leftarrow E^{t_{p}}_{v},v\in\mathcal{V}$

13: $(v,r_{v}),\forall v\in\mathcal{V}$ generates $A_{p}$ by Eqn.4-5

14: $h_{v}(\Delta t)\leftarrow\mathrm{ODESolve}(g(t),h_{v}(0),\Delta t)$,
$\forall v\in\mathcal{V}$

15: return $h_{u}(\Delta t),h_{i}(\Delta t)$

16:end procedure

17:procedure Graph Attention Layer($\bar{h}^{t}_{u},\bar{h}^{t}_{i},u,i,t$)

18: for $v$ in $\left\\{u,i\right\\}$ do

19: Sample $k$ temporal neighbors $N(v,t)$

20: $z^{t}_{v}\leftarrow\bar{h}^{t}_{v}\left|\right|F_{T}(t),$

21:
$Z^{t}_{N(v,t)}\leftarrow\bar{H}^{t}_{N(v,t)}\left|\right|f^{vN(v,t)}\left|\right|F_{T}(t-t_{j})$

22:
$h^{t}_{N(v,t)}\leftarrow\mathrm{Attn}(z^{t}_{v},Z^{t}_{N(v,t)},Z^{t}_{N(v,t)})$

23: $\hat{h}^{t}_{v}\leftarrow f(h^{t}_{N(v,t)}\left|\right|\bar{h}^{t}_{v})$

24: end for

25: return $\hat{h}^{t}_{u},\hat{h}^{t}_{i}$

26:end procedure

###  4.6 Complexity Analysis

The time complexity of our proposed method mainly consists of two portions.

First, for the update module (i.e., continuous inference block), we consider
that both adjacency matrices are stored as sparse matrices. And the runtime of
the ODE solver depends on the length on the time interval (i.e., the end time
of ODE solver) $\Delta t$ and the complexity of the dynamics. Then, the time
complexity of the continuous inference block is $O(\Delta
t\left|\mathcal{E}\right|)$.

Second, for the transform module (i.e., graph attention layer), since the
masked self-attention operation is parallelizable, as suggested by [33]. The
per-batch time complexity of the graph attention layer with $m$ heads can be
expressed as $O(mk)$, where $k$ is the average neighborhood size. Since the
batch is divided by edges, the time complexity of the graph attention layer is
$O(mk\left|\mathcal{E}\right|)$, where $m\ll\left|\mathcal{E}\right|$ and
$k\ll\left|\mathcal{E}\right|$.

Therefore, the time complexity of ConTIG is $O((\Delta
t+mk)\left|\mathcal{E}\right|)\approx O(C\left|\mathcal{E}\right|)$, where $C$
is a constant and is relative to the runtime of the ODE Solver.

##  5 Experiments

In this section, we will utilize four networks to compare the performance of
our model with four static methods and five dynamic methods. We conduct three
main tasks, link prediction, node recommendation and node classification, to
evaluate the influence of introducing temporal information and learn
continuous dynamic of node embedding trajectories.

###  5.1 Experimental Setup

Datasets We evaluate the performance of ConTIG on temporal link prediction,
node recommendation and dynamic node classification tasks with four public
datasets, where three datasets are user-item networks selected by [25] and one
dataset is e-mail network. The statistics of the four datasets are summarized
in Table I.

  * •

Wikipedia222http://snap.stanford.edu/jodie/wikipedia.csv. The dataset
describes the interactions between active users and pages they edit the
contents with unique timestamps and the dynamic labels indicating whether a
user is banned from editing.

  * •

Reddit333http://snap.stanford.edu/jodie/reddit.csv. The dataset describes the
interactions between active users and the posts they submit on subreddits and
the dynamic labels indicating if users are banned from posting.

  * •

Mooc444http://snap.stanford.edu/jodie/mooc.csv. The dataset describes the
interactions between students and MOOC online course, e.g., viewing a video,
submitting an answer, etc.

  * •

CollegeMsg555https://snap.stanford.edu/data/CollegeMsg.html. The dataset is an
online social network at the University of California, and describes the
interaction between users by sending private message at different timestamps.
The dataset is without node label and edge features.

Baselines To evaluate the performance of ConTIG, we compare our method with
state-of-the-art graph embedding methods on both static and dynamic graphs.

Static graph embedding methods: GAE and VGAE [52] learn latent representations
by graph auto-encoder. GraphSAGE [8] and GAT [9] are GNNs that capture the
dependence of graphs via message passing between the nodes of graphs.

Dynamic graph embedding methods: CTDNE [21] defines the temporal random walk
requiring the walks to obey the temporal order. JODIE [25] learns to generate
embedding trajectories of all users and items from temporal interactions by
update and project operations. DyRep [40] use deep recurrent architecture and
attention mechanism to effectively model fine-grained temporal dynamic. TGAT
[23] introduces a self-attention mechanism and a functional time encoding
technique to learn the time-feature interactions. TGN [24] is a generic
inductive framework operating on continuous-time dynamic graphs [10]
represented as a sequence of events.

TABLE I: Statistics of the datasets. Datasets | $\left|\mathcal{V}\right|$ | $\left|\mathcal{E}\right|$ | Feature | Label
---|---|---|---|---
Wikipedia | 9,227 | 157,474 | 172 | 2
Reddit | 10,984 | 672,447 | 172 | 2
Mooc | 7,144 | 411,749 | 0 | 2
CollegeMsg | 1,899 | 59,835 | 0 | 0
TABLE II: ROC AUC(%) and Average Precision(%) for the transductive temporal link prediction on Wikipedia, Reddit, Mooc and CollegeMsg. The means and standard deviations are computed ten runs. Task | Methods | Wikipedia | Reddit | Mooc | CollegeMsg
---|---|---|---|---|---
AUC | AP | AUC | AP | AUC | AP | AUC | AP
Transductive | GAE | $91.47\pm 0.3$ | $91.12\pm 0.1$ | $95.87\pm 1.2$ | $96.57\pm 1.0$ | $87.89\pm 0.6$ | $90.70\pm 0.3$ | $73.15\pm 1.5$ | $70.00\pm 1.17$
VGAE | $82.43\pm 1.6$ | $82.50\pm 4.0$ | $92.70\pm 0.4$ | $91.53\pm 0.7$ | $88.21\pm 0.6$ | $\textbf{91.00}\pm 0.3$ | $74.07\pm 0.9$ | $70.66\pm 1.0$
GraphSAGE | $92.00\pm 0.3$ | $92.34\pm 0.3$ | $97.75\pm 0.1$ | $97.85\pm 0.1$ | $56.17\pm 0.3$ | $60.63\pm 0.2$ | $62.38\pm 1.3$ | $62.48\pm 0.9$
GAT | $92.76\pm 0.5$ | $93.17\pm 0.5$ | $97.90\pm 0.1$ | $97.07\pm 0.1$ | $67.24\pm 0.1$ | $66.66\pm 0.8$ | $78.09\pm 0.5$ | $75.97\pm 0.7$
CTDNE | $82.36\pm 0.7$ | $80.86\pm 0.7$ | $85.32\pm 2.0$ | $87.31\pm 1.4$ | $88.97\pm 2.6$ | $89.27\pm 2.0$ | $81.88\pm 0.7$ | $80.25\pm 0.8$
JODIE | $94.94\pm 0.3$ | $94.65\pm 0.6$ | $97.62\pm 0.2$ | $97.07\pm 0.4$ | $79.75\pm 2.8$ | $74.85\pm 3.1$ | $59.85\pm 6.0$ | $54.50\pm 4.4$
DyRep | $94.22\pm 0.2$ | $94.63\pm 0.2$ | $98.01\pm 0.1$ | $98.05\pm 0.1$ | $80.57\pm 2.1$ | $77.30\pm 2.2$ | $54.75\pm 6.8$ | $51.89\pm 4.8$
TGAT | $94.99\pm 0.3$ | $95.29\pm 0.2$ | $98.07\pm 0.1$ | $98.17\pm 0.1$ | $66.02\pm 1.0$ | $63.82\pm 0.9$ | $81.05\pm 0.6$ | $79.16\pm 0.6$
| TGN | $98.42\pm 0.1$ | $98.50\pm 0.1$ | $98.69\pm 0.1$ | $98.73\pm 0.1$ | $89.07\pm 1.6$ | $86.96\pm 2.1$ | $85.06\pm 5.9$ | $85.38\pm 6.4$
| ConTIG | $\textbf{98.50}\pm 0.2$ | $\textbf{98.62}\pm 0.2$ | $\textbf{98.71}\pm 0.3$ | $\textbf{98.75}\pm 0.3$ | $\textbf{90.34}\pm 1.6$ | $88.87\pm 1.9$ | $\textbf{90.11}\pm 1.2$ | $\textbf{90.54}\pm 1.2$
Inductive | GraphSAGE | $88.60\pm 0.3$ | $88.94\pm 0.5$ | $94.28\pm 0.4$ | $94.51\pm 0.1$ | $53.68\pm 0.4$ | $55.35\pm 0.4$ | $49.64\pm 1.5$ | $51.83\pm 0.8$
GAT | $89.11\pm 0.5$ | $89.82\pm 0.4$ | $94.30\pm 0.4$ | $94.58\pm 0.3$ | $53.43\pm 2.1$ | $54.80\pm 0.9$ | $68.98\pm 1.2$ | $66.22\pm 1.2$
JODIE | $92.75\pm 0.3$ | $93.11\pm 0.4$ | $95.42\pm 0.2$ | $94.50\pm 0.6$ | $81.43\pm 0.8$ | $76.82\pm 1.4$ | $51.59\pm 3.2$ | $50.02\pm 2.2$
DyRep | $91.03\pm 0.3$ | $91.96\pm 0.2$ | $95.79\pm 0.5$ | $95.75\pm 0.5$ | $82.06\pm 1.7$ | $79.17\pm 1.6$ | $49.05\pm 4.1$ | $49.30\pm 2.6$
TGAT | $93.37\pm 0.3$ | $93.86\pm 0.3$ | $96.46\pm 0.1$ | $96.61\pm 0.2$ | $69.09\pm 0.8$ | $67.65\pm 0.7$ | $72.27\pm 0.5$ | $72.53\pm 0.6$
TGN | $97.72\pm 0.1$ | $97.83\pm 0.1$ | $97.54\pm 0.1$ | $97.63\pm 0.1$ | $\textbf{89.03}\pm 1.6$ | $\textbf{86.70}\pm 2.0$ | $78.54\pm 3.9$ | $80.77\pm 3.7$
ConTIG | $\textbf{98.44}\pm 0.2$ | $\textbf{98.41}\pm 0.2$ | $\textbf{98.26}\pm 0.3$ | $\textbf{98.31}\pm 0.2$ | $86.77\pm 2.0$ | $85.44\pm 2.0$ | $\textbf{83.63}\pm 0.9$ | $\textbf{85.37}\pm 0.9$

Parameter Settings For parameter settings, the dimension of both node
representations and time representations are set as 172. The optimizer is Adam
algorithm, learning rate is 0.0001, and dropout probability is 0.1. In
continuous inference block, parameter $\beta$ in adjacency matrix
regularization is 0.95, and the end time of ODE is set as 1.0 instead of the
real interval $\Delta t$. In graph attention layer, the number of selected
neighbors $k$ is set as 15 and heads $M$ in attention is set as 2, and the
negative sampling distribution $P(i)$ is uniform distribution.

Settings for Baselines For static baselines, we adopt the same baseline
training procedures as in [23]. We refer to the PyTorch geometric library for
implementing the GAE and VGAE baselines, and develop off-the-shelf
implementation for GraphSAGE and GAT by referencing their original
implementations to accommodate the temporal setting and incorporate edges
features. For dynamic baselines, we use the open-source implementations for
CTDNE, and use the source code released by the authors to implementing TGAT
and TGN, where we implement JODIE and DyRep in the version of TGN in PyTorch.

Implementation All our experiments are implemented on a 32g-MEM Ubuntu 20.04
server with Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz and NVidia(R) 2080Ti
GPUs. All code is implemented in Pytorch version 1.5.1.

###  5.2 Temporal Link Prediction

The goal of the temporal link prediction task is to predict the probability
that there exists a link between the two nodes given two nodes and a certain
timestamp in the future. For this task, we evaluate our method on both the
transductive and inductive setting. In the transductive task, we predict the
links between nodes observed during training. In the inductive task, we
predict the links between new nodes which have not been observed in the past.
Our model is tuned on the validation set and we report the average precision
(AP) on the test set. We divide the training, validation, and testing sets
into a 70%-15%-15% split. After the division, the nodes that do not appear in
the training set are considered as new nodes. The model is trained by temporal
link prediction.

The results comparison between our method and baseline methods in temporal
link prediction tasks are shown in Table II. We observe that: 1) static graph
embedding methods including GAE, VGAE, GraphSAGE and GAT, perform worse than
those baselines modeling temporal interaction information. Because most of the
interactions in the real-world networks are time-stamped; 2) among the dynamic
graph embedding works, both temporal neighbor information (i.e., CTDNE, TGAT)
and latest interaction information (i.e., JODIE, DyRep) methods perform worse
than fusing methods (i.e., TGN, ConTIG); and 3) ConTIG achieves the best
prediction performance on the datasets Wikipedia, Reddit, and CollegeMsg for
both transductive and inductive settings, and the second best performance on
Mooc for the inductive task. This observation demonstrates the advantage of
our proposed method compared to existing methods. In fact, by considering the
inherent node properties and modeling the three important factors (i.e.,
latest interaction, neighbor features and inherent characteristics) in
temporal interaction graphs, our method can capture the complex non-linear
dynamics of node representations effectively, thereby achieving superb
performance.

###  5.3 Temporal Node Recommendation

(a) Recall@K on Wikipedia (b) Recall@K on Reddit (c) Recall@K on Mooc

Figure 3: Recall@K for the transductive temporal node recommendation on
Wikipedia, Reddit and Mooc.

The goal of temporal node recommendation task is to predict the top-K possible
neighbors of node $u$ at $t$ given the node $u$ and future timestamp $t$. This
task is also used to evaluate the performance of temporal network embedding
methods. For this task, we evaluate all methods on transductive setting, each
method outputs the user $u$’s preference scores over all the items at time in
test set. We sort scores in a descending order and record the rank of the
paired node $u$ and $i$. We evaluate the task on three user-item networks
(i.e., Wikipedia, Reddit and Mooc), where CollegeMsg dataset is not included
because it is not a user-item network. Our evaluation metric in this task is
Recall@K, where $K\in\left\\{5,10,15,20\right\\}$.

The results comparison between our method and baseline methods in temporal
node recommendation task is shown in Fig.3, showing that our model ConTIG
perform better than all the baselines. Compared with the best competitors
(i.e., TGN), the recommendation performance of ConTIG improves by 110.51% in
terms of Recall@5 on Mooc. On Wikipedia and Reddit, the improvement is 3.25%
and 13.13% with respect to Recall@5. These significant improvements verify
that the differential equation fused with three factors (i.e., latest
interaction, neighbor features and inherent characteristics) proposed in
ConTIG is capable of learning the trend of node state trajectories in the
network. Additionally, the significant improvement of ConTIG benefits from the
combination of continuous node state modeling in update module and dynamic
sub-graph structure capturing in transform module on node embeddings, which is
good for the down-stream node recommendation task.

###  5.4 Dynamic Node Classification

TABLE III: ROC AUC(%) for the transductive dynamic node classification on Wikipedia, Reddit and Mooc. The means and standard deviations are computed ten runs.  | Wikipedia | Reddit | Mooc
---|---|---|---
CTDNE | $84.86\pm 1.5$ | $54.38\pm 7.5$ | $71.84\pm 1.0$
JODIE | $84.40\pm 0.9$ | $61.51\pm 1.2$ | $70.03\pm 0.5$
DyRep | $83.25\pm 0.5$ | $60.86\pm 1.7$ | $64.64\pm 1.4$
TGAT | $84.41\pm 1.5$ | $65.98\pm 1.6$ | $65.79\pm 0.5$
TGN | $\textbf{87.56}\pm 0.7$ | $69.78\pm 0.8$ | $63.93\pm 0.3$
ConTIG | $87.13\pm 0.6$ | $\textbf{69.99}\pm 0.5$ | $\textbf{73.56}\pm 0.4$

The goal of dynamic node classification task is to predict the state label of
user given the user, item and future timestamp. For this task, we evaluate our
method on transductive setting, predicting the state labels of users who have
been observed during training. We evaluate the task on three datasets with
dynamic node labels (i.e., Wikipedia, Reddit and Mooc), where CollegeMsg
dataset is not included because there is no node label. Specifically, we train
a decoder after the the model trained by the temporal link prediction task.
Our evaluation metric in this task is the area under the ROC curve (AUC).

The results comparison between our method and baseline methods in dynamic node
classification tasks are shown in Table III. Again, our algorithm achieves the
best or comparable performance comparing with existing dynamic graph embedding
approaches, demonstrating its effectiveness for the down-stream node
classification task.

###  5.5 Performance on Long-interval Interactions

(a) Wikipedia

(b) CollegeMsg

Figure 4: Average Precision(%) for both the transductive and inductive temporal link prediction on five interaction sets of Wikipedia and CollegeMsg, divided by quantiles of time interval of the interaction. TABLE IV: Average Precision(%) for both the transductive and inductive temporal link prediction on Wikipedia and CollegeMsg.  | Wikipedia | Reddit | Mooc | CollegeMsg
---|---|---|---|---
Transductive | Inductive | Transductive | Inductive | Transductive | Inductive | Transductive | Inductive
ConTIG w/o transform | $95.99$ | $96.76$ | $98.25$ | $97.88$ | $82.33$ | $75.22$ | $87.66$ | $67.65$
ConTIG w/o update | $95.19$ | $95.03$ | $95.85$ | $93.36$ | $75.51$ | $73.10$ | $82.78$ | $80.90$
ConTIG w/o adaptive | $98.66$ | $98.36$ | $98.75$ | $98.15$ | $68.56$ | $62.42$ | $87.72$ | $84.23$
ConTIG w/o latest | $98.50$ | $98.45$ | $98.45$ | $98.79$ | $90.25$ | $86.76$ | $90.66$ | $85.42$
ConTIG w/o neighbor | $98.38$ | $98.25$ | $98.02$ | $96.95$ | $84.03$ | $79.57$ | $89.08$ | $83.13$
ConTIG w/o inherent | 98.75 | $98.56$ | $98.79$ | $98.35$ | $86.74$ | $84.27$ | $90.66$ | $82.72$
ConTIG | $98.61$ | 98.58 | 98.79 | 98.43 | 90.56 | 87.31 | 91.40 | 86.42

The goal of this task is to observe the link prediction performance of our
method for interaction sets with different time intervals. To categorize the
interactions, first, we calculate the time interval $\Delta t$ between each
interaction and its latest interaction as mentioned in Section 3, and describe
the distribution of the time intervals of interactions. Then, we equally
divide the interactions into five sets according to the four quantiles: 20%,
40%, 60%, 80%, where the quantiles are cut points dividing the interactions
into continuous intervals with equal probabilities in terms of the intervals
$\Delta t$ of interactions. For example, the interactions in 0 - 20% set are
of shorter intervals, while the interactions in 80-100% set are of longer
intervals. Finally, we calculate the AP score for each set.

Our results of comparison between our method and dynamic graph baselines in
temporal link prediction task on five interaction sets of Wikipedia and
CollegeMsg are shown in Fig. 4. Comparing the AP results in each set, we find
that our method outperforms TGN, especially in long-interval interactions
(e.g., the frequency interval 60-80% and 80-100%), demonstrating the
superiority of ConTIG in capturing continuous dynamics of node
representations. And we argue that the long-interval link prediction is more
beneficial to practical applications, because quite a few users are always
inactive on social networks, citation networks and other user-item interaction
systems.

###  5.6 Ablation Study

To further investigate the effect of each component in our model, we compare
ConTIG with its variants as follows:

  * •

ConTIG w/o transform: ConTIG without the transform module (i.e. graph
attention layer).

  * •

ConTIG w/o update: ConTIG without the update module (i.e. continuous inference
block).

  * •

ConTIG w/o adaptive: ConTIG without adaptively fusing three factors in update
module. We replace it by directly adding the three factors in the update
module.

  * •

ConTIG w/o latest: ConTIG without the latest interaction factor in update
module.

  * •

ConTIG w/o neighbor: ConTIG without the neighbor features factor in update
module.

  * •

ConTIG w/o inherent: ConTIG without the inherent characteristics factor in
update module.

Table IV shows the AP results of each model. ConTIG performs better than
ConTIG w/o transform and ConTIG w/o update by a large margin, which
demonstrates the effectiveness of the main modules of our model. Especially,
the introduction of the continuous update module significantly improves the
results, indicating that the latest knowledge between two consecutive
interactions is an essential feature to learn the node representations. By
further modeling and aggregating the historical interaction information of
nodes, ConTIG consistently improves the performance, showing the importance of
the historical interaction information.

Specifically, we investigate the effect of each component in update module. By
removing the adaptive fusion approach, the performance of ConTIG w/o adaptive
degrades obviously, pointing out that the adaptive fusion is a key factor to
the success of the ODE solver in update module. It helps the network to focus
on the most correlated factor to update the node state, and adaptively fuses
three factors in a data-dependent way. Besides, ConTIG outperforms ConTIG w/o
latest, ConTIG w/o neighbor and ConTIG w/o inherent by a small margin, which
indicates that the three factors is of great importance for learning the
change of node state in the update module. In particular, by comparing ConTIG
with ConTIG w/o latest, ConTIG w/o neighbor and ConTIG w/o inherent,
respectively, we observe that the neighbor features contributes most to the
performance in all datasets, which indicates the importance of neighbors in
the latest changing of node states. Simultaneously, for ConTIG w/o inherent,
we observe that it has obvious effects on the datasets without edges features
(i.e., Mooc and CollegeMsg), although it has tiny effects on the datasets with
edge features (i.e., Wikipedia and Reddit).

###  5.7 Parameter Sensitivity

(a) end time in ODE $\Delta t$

(b) number of neighbors $k$

(c) number of heads $M$

(d) node embedding dimension $d$

(e) time embedding dimension $d^{T}$

Figure 5: Average Precision(%) for both transductive and inductive temporal
link prediction on Wikipedia and Mooc with different hyper-parameters. Figure
6: Mean time over one epoch during the entire ConTIG training on Wikipedia
with different $\left|\mathcal{E}\right|$ for training.

There are several hyper-parameters in our proposed method. It is necessary to
analyze how these parameters influence the performance of our model in
temporal link prediction task on the above datasets. In detail, these
parameters include the end time $\Delta t$ in continuous inference block, the
number of neighbors $k$, heads $M$ in graph attention layer, node embedding
dimension $d$ and time embedding dimension $d^{T}$. We choose different values
for them and use the AP score to evaluate their performance.

End time in continuous inference block (See Section 4.3). An important
parameter is how long used to update the node states in the continuous
inference block. In our experiment the $\Delta t$ ranges from 0.6 to 1.6 and
the node and time embedding dimension is fixed to 172. As shown in Fig.5a, as
$\Delta t$ is larger, the AP score first increases and then decreases,
demonstrating that more time for learning node changes between two consecutive
interactions could yield better performance. However, when the updating time
is very long, it would make the current node over rely on the latest
information, and forget the historical information, which hinders the
performance.

Number of neighbors (See Section 4.4). The influence of the number of
neighbors will then be evaluated, which control the amount of historical
interactive information is considered in the transform module. As shown in
Fig.5b, in general, as the number of neighborhood $k$ becomes larger, the
model could achieve better performance because more historical information is
considered. However, when the number of neighborhood is very large, it would
introduce useless or overdue information into the learning process and thus
degrade the performance.

Heads in graph attention layer (See Section LABEL:sec.4.4). Fig.5c show the AP
results under different number of heads $M$ in the multi-head attention. We
observed that the AP score first increases and then decreases or stabilize,
and there is a relatively stable and good performance at two-head attention.
It means that less-head attention will make the results unstable and offset,
but attention with many head may pay attention to useless or unrelated
information.

Node embedding dimension. The influence of different node embedding dimensions
16; 64; 128; 172; 256 on our model is shown in Fig. 5d. We discover the same
trend on two datasets that as the dimension rises from a small value, the
performance of ConTIG will improve rapidly and it becomes relative stable
while the size is large. The potential reason is that a higher dimension
enables the model to learn more information in the latent space. However, it
would make computational consumption become very large, so we choose a
reasonable dimension with the best performance (i.e., 172).

Time embedding dimension. The influence of different time embedding dimensions
16; 64; 128; 172; 256 on our model is shown in Fig. 5e. We observe the similar
results with node embedding dimension, as $d^{T}$ is larger, the AP score
first increases and then tends to be stable. Different from node embedding
dimension, there is a fluctuation in the high-dimension time embedding on
Wikipedia and low-dimension time embedding on Mooc, which means high dimension
and low dimensions may both hinder the performance, and shows the importance
of choosing a reasonable time embedding dimension (i.e., 172).

###  5.8 Complexity Evaluation

Here, we examine how the training time of ConTIG depends on the number of
edges $\left|\mathcal{E}\right|$ which are used for training. We record the
runtimes of ConTIG for training one epoch on the Wikipedia dataset using the
best parameters in Section 5.7. Fig.6 shows that the entire runtime for one-
epoch training is close to linear with $\left|\mathcal{E}\right|$. This
evaluation demonstrates our method’s advantage of being scalable to process
long edge streams.

##  6 Conclusion

In this paper, we present ConTIG, a novel representation learning method to
capture the continuous dynamic of node embedding trajectories by identifying
three-fold factors (i.e., latest interaction, neighbor features and inherent
characteristics). ConTIG contains two modules: a update module to learn the
node embedding trajectories and a transform module to generate the future
representations according to the historical interaction information.
Experiments results demonstrate that ConTIG achieves state-of-the-art
performances, especially for long-interval interactions on temporal link
prediction tasks. In the future, besides node state trajectory, we plan to pay
more attention to community evolution, exploring the impact of community on
individuals during the graph evolution.

## Acknowledgments

The research is supported by Natural Science Foundation of China (61872306),
Xiamen Science and Technology Bureau (3502Z20193017) and Fundamental Research
Funds for the Central Universities (20720200031).

## References

  * [1] W. L. Hamilton, R. Ying, and J. Leskovec, “Representation learning on graphs: Methods and applications,” _arXiv preprint arXiv:1709.05584_ , 2017.
  * [2] P. Cui, X. Wang, J. Pei, and W. Zhu, “A survey on network embedding,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 31, no. 5, pp. 833–852, 2018.
  * [3] B. Perozzi, R. Al-Rfou, and S. Skiena, “Deepwalk: Online learning of social representations,” in _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2014, pp. 701–710.
  * [4] J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei, “Line: Large-scale information network embedding,” in _Proceedings of the 24th international conference on world wide web_ , 2015, pp. 1067–1077.
  * [5] A. Grover and J. Leskovec, “node2vec: Scalable feature learning for networks,” in _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2016, pp. 855–864.
  * [6] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip, “A comprehensive survey on graph neural networks,” _IEEE transactions on neural networks and learning systems_ , vol. 32, no. 1, pp. 4–24, 2020.
  * [7] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017\. [Online]. Available: https://openreview.net/forum?id=SJU4ayYgl
  * [8] W. L. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA_ , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 1024–1034. [Online]. Available: http://papers.nips.cc/paper/6703-inductive-representation-learning-on-large-graphs
  * [9] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018. [Online]. Available: https://openreview.net/forum?id=rJXMpikCZ
  * [10] S. M. Kazemi, R. Goel, K. Jain, I. Kobyzev, A. Sethi, P. Forsyth, and P. Poupart, “Representation learning for dynamic graphs: A survey.” _Journal of Machine Learning Research_ , vol. 21, no. 70, pp. 1–73, 2020\.
  * [11] J. Skarding, B. Gabrys, and K. Musial, “Foundations and modelling of dynamic networks using dynamic graph neural networks: A survey,” _arXiv preprint arXiv:2005.07496_ , 2020.
  * [12] G. Xue, M. Zhong, J. Li, J. Chen, C. Zhai, and R. Kong, “Dynamic network embedding survey,” _arXiv preprint arXiv:2103.15447_ , 2021.
  * [13] Y. Zhao, X. Wang, H. Yang, L. Song, and J. Tang, “Large scale evolving graphs with burst detection,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ , S. Kraus, Ed. ijcai.org, 2019, pp. 4412–4418. [Online]. Available: https://doi.org/10.24963/ijcai.2019/613
  * [14] A. Pareja, G. Domeniconi, J. Chen, T. Ma, T. Suzumura, H. Kanezashi, T. Kaler, T. B. Schardl, and C. E. Leiserson, “Evolvegcn: Evolving graph convolutional networks for dynamic graphs.” in _AAAI_ , 2020, pp. 5363–5370.
  * [15] S. Spasov, A. Di Stefano, P. Liò, and J. Tang, “Grade: Graph dynamic embedding,” _arXiv preprint arXiv:2007.08060_ , 2020.
  * [16] J. Liu, C. Xu, C. Yin, W. Wu, and Y. Song, “K-core based temporal graph convolutional network for dynamic graphs,” _IEEE Transactions on Knowledge and Data Engineering_ , 2020.
  * [17] M. Gong, S. Ji, Y. Xie, Y. Gao, and A. Qin, “Exploring temporal information for dynamic network embedding,” _IEEE Transactions on Knowledge and Data Engineering_ , 2020.
  * [18] P. Jiao, X. Guo, X. Jing, D. He, H. Wu, S. Pan, M. Gong, and W. Wang, “Temporal network embedding for link prediction via vae joint attention mechanism,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2021.
  * [19] L. Qu, H. Zhu, Q. Duan, and Y. Shi, “Continuous-time link prediction via temporal dependent graph neural network,” in _Proceedings of The Web Conference 2020_ , 2020, pp. 3026–3032.
  * [20] Y. Li, D. S. H. Tam, S. Xie, X. Liu, Q. F. Ying, W. C. Lau, D. M. Chiu, and S. Z. Chen, “Gtea: Representation learning for temporal interaction graphs via edge aggregation,” _arXiv preprint arXiv:2009.05266_ , 2020.
  * [21] G. H. Nguyen, J. B. Lee, R. A. Rossi, N. K. Ahmed, E. Koh, and S. Kim, “Continuous-time dynamic network embeddings,” in _Companion Proceedings of the The Web Conference 2018_ , 2018, pp. 969–976.
  * [22] N. Noorshams, S. Verma, and A. Hofleitner, “Ties: Temporal interaction embeddings for enhancing social media integrity at facebook,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 3128–3135.
  * [23] D. Xu, C. Ruan, E. Körpeoglu, S. Kumar, and K. Achan, “Inductive representation learning on temporal graphs,” in _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020\. [Online]. Available: https://openreview.net/forum?id=rJeW1yHYwH
  * [24] E. Rossi, B. Chamberlain, F. Frasca, D. Eynard, F. Monti, and M. M. Bronstein, “Temporal graph networks for deep learning on dynamic graphs,” _CoRR_ , vol. abs/2006.10637, 2020. [Online]. Available: https://arxiv.org/abs/2006.10637
  * [25] S. Kumar, X. Zhang, and J. Leskovec, “Predicting dynamic embedding trajectory in temporal interaction networks,” in _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_, A. Teredesai, V. Kumar, Y. Li, R. Rosales, E. Terzi, and G. Karypis, Eds., 2019, pp. 1269–1278.
  * [26] Z. Zhang, J. Bu, M. Ester, J. Zhang, C. Yao, Z. Li, and C. Wang, “Learning temporal interaction graph embedding via coupled memory networks,” in _Proceedings of The Web Conference 2020_ , 2020, pp. 3049–3055.
  * [27] K. Oono and T. Suzuki, “Graph neural networks exponentially lose expressive power for node classification,” in _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020. [Online]. Available: https://openreview.net/forum?id=S1ldO2EFPr
  * [28] L. A. C. Xhonneux, M. Qu, and J. Tang, “Continuous graph neural networks,” in _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 10 432–10 441. [Online]. Available: http://proceedings.mlr.press/v119/xhonneux20a.html
  * [29] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, “Neural ordinary differential equations,” in _Advances in neural information processing systems_ , 2018, pp. 6571–6583.
  * [30] M. Belkin and P. Niyogi, “Laplacian eigenmaps and spectral techniques for embedding and clustering.” in _Nips_ , vol. 14, no. 14, 2001, pp. 585–591.
  * [31] A. Ahmed, N. Shervashidze, S. Narayanamurthy, V. Josifovski, and A. J. Smola, “Distributed large-scale natural graph factorization,” in _Proceedings of the 22nd international conference on World Wide Web_ , 2013, pp. 37–48.
  * [32] S. Pan, R. Hu, S.-f. Fung, G. Long, J. Jiang, and C. Zhang, “Learning graph embedding with adversarial training methods,” _IEEE transactions on cybernetics_ , vol. 50, no. 6, pp. 2475–2487, 2020.
  * [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [34] U. Singer, I. Guy, and K. Radinsky, “Node embedding over temporal graphs,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ , S. Kraus, Ed. ijcai.org, 2019, pp. 4605–4612. [Online]. Available: https://doi.org/10.24963/ijcai.2019/640
  * [35] A. Sankar, Y. Wu, L. Gou, W. Zhang, and H. Yang, “Dysat: Deep neural representation learning on dynamic graphs via self-attention networks,” in _Proceedings of the 13th International Conference on Web Search and Data Mining_ , 2020, pp. 519–527.
  * [36] M. Yang, M. Zhou, M. Kalander, Z. Huang, and I. King, “Discrete-time temporal network embedding via implicit hierarchical learning in hyperbolic space,” in _Proc. of 2021 ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD’21)_ , 2021.
  * [37] X. Guo, B. Zhou, and S. Skiena, “Subset node representation learning over large dynamic graphs,” in _Proc. of 2021 ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD’21)_ , 2021.
  * [38] P. Goyal, N. Kamra, X. He, and Y. Liu, “Dyngem: Deep embedding method for dynamic graphs,” _arXiv preprint arXiv:1805.11273_ , 2018.
  * [39] D. Xu, J. Liang, W. Cheng, H. Wei, H. Chen, and X. Zhang, “Transformer-style relational reasoning with dynamic memory updating for temporal network modeling,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 5, 2021, pp. 4546–4554.
  * [40] R. Trivedi, M. Farajtabar, P. Biswal, and H. Zha, “Dyrep: Learning representations over dynamic graphs,” in _International Conference on Learning Representations_ , 2019.
  * [41] Y. Lu, X. Wang, C. Shi, P. S. Yu, and Y. Ye, “Temporal network embedding with micro-and macro-dynamics,” in _Proceedings of the 28th ACM international conference on information and knowledge management_ , 2019, pp. 469–478.
  * [42] L. Wang, X. Chang, S. Li, Y. Chu, H. Li, W. Zhang, X. He, L. Song, J. Zhou, and H. Yang, “Tcl: Transformer-based dynamic graph modelling via contrastive learning,” _arXiv preprint arXiv:2105.07944_ , 2021.
  * [43] D. Zhou, L. Zheng, J. Han, and J. He, “A data-driven graph generative model for temporal interaction networks,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 401–411.
  * [44] M. Liu, Z. Tu, X. Xu, and Z. Wang, “Learning representation over dynamic graph using aggregation-diffusion mechanism,” _arXiv preprint arXiv:2106.01678_ , 2021.
  * [45] A. Paranjape, A. R. Benson, and J. Leskovec, “Motifs in temporal networks,” in _Proceedings of the tenth ACM international conference on web search and data mining_ , 2017, pp. 601–610.
  * [46] Y. Wang, Y.-Y. Chang, Y. Liu, J. Leskovec, and P. Li, “Inductive representation learning in temporal networks via causal anonymous walks,” _arXiv preprint arXiv:2101.05974_ , 2021.
  * [47] D. Fu, D. Zhou, and J. He, “Local motif clustering on time-evolving graphs,” in _Proceedings of the 26th ACM SIGKDD International conference on knowledge discovery & data mining_, 2020, pp. 390–400.
  * [48] C. Zang and F. Wang, “Neural dynamics on complex networks,” in _KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020_ , R. Gupta, Y. Liu, J. Tang, and B. A. Prakash, Eds. ACM, 2020, pp. 892–902. [Online]. Available: https://doi.org/10.1145/3394486.3403132
  * [49] Z. Ding, Z. Han, Y. Ma, and V. Tresp, “Temporal knowledge graph forecasting with neural ode,” _arXiv preprint arXiv:2101.05151_ , 2021.
  * [50] Z. Huang, Y. Sun, and W. Wang, “Coupled graph ode for learning interacting system dynamics,” in _Proc. of 2021 ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD’21)_ , 2021.
  * [51] D. Xu, C. Ruan, E. Körpeoglu, S. Kumar, and K. Achan, “Self-attention with functional time representation learning,” in _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 15 889–15 899. [Online]. Available: http://papers.nips.cc/paper/9720-self-attention-with-functional-time-representation-learning
  * [52] T. N. Kipf and M. Welling, “Variational graph auto-encoders,” _CoRR_ , vol. abs/1611.07308, 2016. [Online]. Available: http://arxiv.org/abs/1611.07308