* Borgwardt et al. (2005) Karsten M. Borgwardt, Cheng Soon Ong, and et al. 2005\.  Protein function prediction via graph kernels. In _ISMB 2005_. 47–56.
  * Bouhenni et al. (2022) Sarra Bouhenni, Saïd Yahiaoui, and et al. 2022\.  A Survey on Distributed Graph Pattern Matching in Massive Graphs.  _ACM Comput. Surv._ 54, 2 (2022), 36:1–36:35.
  * Brandes (2001) Ulrik Brandes. 2001.  A Faster Algorithm for Betweenness Centrality, In Journal of Mathematical Sociology.  _Journal of Mathematical Sociology_ 25, 163–177.
  * Brin and Page (1998) Sergey Brin and Lawrence Page. 1998\.  The Anatomy of a Large-Scale Hypertextual Web Search Engine.  _Comput. Networks_ 30 (1998), 107–117.
  * Broder et al. (2000) Andrei Z. Broder, Ravi Kumar, and et al. 2000\.  Graph structure in the Web.  _Comput. Networks_ 33, 1-6 (2000), 309–320.
  * Busato and Bombieri (2015) Federico Busato and Nicola Bombieri. 2015\.  BFS-4K: An Efficient Implementation of BFS for Kepler GPU Architectures.  _IEEE Trans. Parallel Distributed Syst._ 26, 7 (2015), 1826–1838.
  * Busato and Bombieri (2016) Federico Busato and Nicola Bombieri. 2016\.  An Efficient Implementation of the Bellman-Ford Algorithm for Kepler GPU Architectures.  _IEEE Trans. Parallel Distributed Syst._ 27, 8 (2016), 2222–2233.
  * Cao et al. (2022) Huanqi Cao, Yuanwei Wang, and et al. 2022\.  Scaling graph traversal to 281 trillion edges with 40 million cores. In _PPoPP 2022_. 234–245.
  * Cao, Fineman and Russell (2021) Nairen Cao, Jeremy T. Fineman, and et al. 2021\.  Brief Announcement: An Improved Distributed Approximate Single Source Shortest Paths Algorithm. In _PODC 2021_. 493–496.
  * Carbone et al. (2015) Paris Carbone, Asterios Katsifodimos, and et al. 2015\.  Apache flink: Stream and batch processing in a single engine.  _Bull. Tech. Comm. Learn. Technol._ 38, 4 (2015).
  * Censor-Hillel et al. (2019) Keren Censor-Hillel, Eldar Fischer, and et al. 2019\.  Fast distributed algorithms for testing graph properties.  _Distributed Comput._ 32, 1 (2019), 41–57.
  * Censor-Hillel, Gall and Leitersdorf (2020) Keren Censor-Hillel, François Le Gall, and et al. 2020\.  On Distributed Listing of Cliques. In _PODC 2020_. 474–482.
  * Censor-Hillel, Leitersdorf and Vulakh (2022) Keren Censor-Hillel, Dean Leitersdorf, and et al. 2022\.  Deterministic Near-Optimal Distributed Listing of Cliques. In _PODC 2022_. 271–280.
  * Chakaravarthy et al. (2017) Venkatesan T. Chakaravarthy, Fabio Checconi, and et al. 2017\.  Scalable Single Source Shortest Path Algorithms for Massively Parallel Systems.  _IEEE Trans. Parallel Distributed Syst._ (2017), 2031–2045.
  * Chan, Sozio and Sun (2021) T.-H. Hubert Chan, Mauro Sozio, and et al. 2021\.  Distributed approximate k-core decomposition and min-max edge orientation: Breaking the diameter barrier.  _J. Parallel Distributed Comput._ 147 (2021), 87–99.
  * Chang et al. (2021) Yi-Jun Chang, Seth Pettie, and et al. 2021\.  Near-optimal Distributed Triangle Enumeration via Expander Decompositions.  _J. ACM_ 68, 3 (2021), 21:1–21:36.
  * Chang, Pettie and Zhang (2019) Yi-Jun Chang, Seth Pettie, and et al. 2019\.  Distributed Triangle Detection via Expander Decomposition. In _SODA 2019_. 821–840.
  * Chang and Saranurak (2019) Yi-Jun Chang and Thatchaphol Saranurak. 2019\.  Improved Distributed Expander Decomposition and Nearly Optimal Triangle Enumeration. In _PODC 2019_. 66–73.
  * Chapman et al. (2010) Barbara Chapman, Tony Curtis, and et al. 2010\.  Introducing OpenSHMEM: SHMEM for the PGAS community. In _PGAS 2010_. 1–3.
  * Charles et al. (2005) Philippe Charles, Christian Grothoff, and et al. 2005\.  X10: an object-oriented approach to non-uniform cluster computing.  _Acm Sigplan Notices_ 40, 10 (2005), 519–538.
  * Chechik and Mukhtar (2019) Shiri Chechik and Doron Mukhtar. 2019\.  Optimal Distributed Coloring Algorithms for Planar Graphs in the LOCAL model. In _SODA 2019_. 787–804.
  * Chechik and Mukhtar (2022) Shiri Chechik and Doron Mukhtar. 2022\.  Single-source shortest paths in the CONGEST model with improved bounds.  _Distributed Comput._ 35, 4 (2022), 357–374.
  * Chen et al. (2018) Hongzhi Chen, Miao Liu, and et al. 2018\.  G-Miner: an efficient task-oriented graph mining system. In _EuroSys 2018_. 32:1–32:12.
  * Chen and Li (2021) Jie Chen and Xiang Li. 2021.  A Minimal Memory Game-Based Distributed Algorithm to Vertex Cover of Networks. In _ISCAS 2021_. 1–5.
  * Chen and Li (2023) Jie Chen and Xiang Li. 2023.  Toward the minimum vertex cover of complex networks using distributed potential games.  _Sci. China Inf. Sci._ 66, 1 (2023).
  * Chen and Qian (2023) Jingji Chen and Xuehai Qian. 2023\.  Khuzdul: Efficient and Scalable Distributed Graph Pattern Mining Engine. In _ASPLOS 2023_. 413–426.
  * Chen, Chou and Chen (2014) Pei-Ling Chen, Chung-Kuang Chou, and et al. 2014\.  Distributed algorithms for k-truss decomposition. In _IEEE BigData 2014_. 471–480.
  * Chen et al. (2021) Xuhao Chen, Tianhao Huang, and et al. 2021\.  FlexMiner: A Pattern-Aware Accelerator for Graph Pattern Mining. In _ISCA 2021_. 581–594.
  * Clauset, Shalizi and Newman (2009) Aaron Clauset, Cosma Rohilla Shalizi, and et al. 2009\.  Power-Law Distributions in Empirical Data.  _SIAM Rev._ 51, 4 (2009), 661–703.
  * Cosulschi et al. (2015) Mirel Cosulschi, Mihai Gabroveanu, and et al. 2015\.  Scaling Up a Distributed Computing Of Similarity Coefficient with Mapreduce.  _Int. J. Comput. Sci. Appl._ 12, 2 (2015), 81–98.
  * Crescenzi, Fraigniaud and Paz (2020) Pierluigi Crescenzi, Pierre Fraigniaud, and et al. 2020\.  Simple and Fast Distributed Computation of Betweenness Centrality. In _INFOCOM 2020_. 337–346.
  * Czarnul, Proficz and Drypczewski (2020) Paweł Czarnul, Jerzy Proficz, and et al. 2020\.  Survey of methodologies, approaches, and challenges in parallel programming using high-performance computing systems.  _Scientific Programming_ 2020 (2020), 1–19.
  * Dagum and Menon (1998) Leonardo Dagum and Ramesh Menon. 1998\.  OpenMP: an industry standard API for shared-memory programming.  _IEEE computational science and engineering_ 5, 1 (1998), 46–55.
  * Dai et al. (2019) Guohao Dai, Tianhao Huang, and et al. 2019\.  GraphH: A Processing-in-Memory Architecture for Large-Scale Graph Processing.  _IEEE Trans. Comput. Aided Des. Integr. Circuits Syst._ 38, 4 (2019), 640–653.
  * Dani et al. (2023) Varsha Dani, Aayush Gupta, and et al. 2023\.  Wake up and join me! An energy-efficient algorithm for maximal matching in radio networks.  _Distributed Comput._ 36, 3 (2023), 373–384.
  * Dean and Ghemawat (2004) Jeffrey Dean and Sanjay Ghemawat. 2004\.  MapReduce: Simplified Data Processing on Large Clusters. In _OSDI 2004_. 137–150.
  * Dominguez-Sal et al. (2010) David Dominguez-Sal, Norbert Martinez-Bazan, and et al. 2010\.  A discussion on the design of graph database benchmarks. In _Technology Conference on Performance Evaluation and Benchmarking_. 25–40.
  * Elkin (2020) Michael Elkin. 2020.  Distributed Exact Shortest Paths in Sublinear Time.  _J. ACM_ 67, 3 (2020), 15:1–15:36.
  * Even, Medina and Ron (2015) Guy Even, Moti Medina, and et al. 2015\.  Distributed Maximum Matching in Bounded Degree Graphs. In _Proceedings of the 2015 International Conference on Distributed Computing and Networking, ICDCN 2015_. 18:1–18:10.
  * Fan et al. (2021) Wenfei Fan, Tao He, and et al. 2021\.  GraphScope: A Unified Engine For Big Graph Processing.  _Proc. VLDB Endow._ 14, 12 (2021), 2879–2892.
  * Feng et al. (2018) Xing Feng, Lijun Chang, and et al. 2018\.  Distributed computing connected components with linear communication cost.  _Distributed Parallel Databases_ 36, 3 (2018), 555–592.
  * Forster and Nanongkai (2018) Sebastian Forster and Danupon Nanongkai. 2018\.  A Faster Distributed Single-Source Shortest Paths Algorithm. In _FOCS 2018_. 686–697.
  * Forum (1994) Message P Forum. 1994.  MPI: A message-passing interface standard.
  * Fournet et al. (2002) Cédric Fournet, Fabrice Le Fessant, and et al. 2002\.  JoCaml: A language for concurrent distributed and mobile programming.  In _International School on Advanced Functional Programming_. 129–158.
  * Fraigniaud and Olivetti (2017) Pierre Fraigniaud and Dennis Olivetti. 2017\.  Distributed Detection of Cycles. In _Proceedings of the 29th ACM Symposium on Parallelism in Algorithms and Architectures, SPAA 2017_. 153–162.
  * Gabow (2017) Harold N Gabow. 2017.  The weighted matching approach to maximum cardinality matching.  _Fundamenta Informaticae_ 154, 1-4 (2017), 109–130.
  * Gallager, Humblet and Spira (1983) Robert G. Gallager, Pierre A. Humblet, and et al. 1983\.  A Distributed Algorithm for Minimum-Weight Spanning Trees.  _ACM Trans. Program. Lang. Syst._ 5, 1 (1983), 66–77.
  * Garay, Kutten and Peleg (1998) Juan A. Garay, Shay Kutten, and et al. 1998\.  A Sublinear Time Distributed Algorithm for Minimum-Weight Spanning Trees.  _SIAM J. Comput._ 27, 1 (1998), 302–316.
  * Ghaffari et al. (2018) Mohsen Ghaffari, Andreas Karrenbauer, and et al. 2018\.  Near-Optimal Distributed Maximum Flow.  _SIAM J. Comput._ 47, 6 (2018), 2078–2117.
  * Ghosh (2022) Sayan Ghosh. 2022.  Improved Distributed-memory Triangle Counting by Exploiting the Graph Structure. In _HPEC 2022_. 1–6.
  * Ghosh and Halappanavar (2020) Sayan Ghosh and Mahantesh Halappanavar. 2020\.  TriC: Distributed-memory Triangle Counting by Exploiting the Graph Structure. In _HPEC 2020_. 1–6.
  * Ghosh et al. (2018) Sayan Ghosh, Mahantesh Halappanavar, and et al. 2018\.  Distributed Louvain Algorithm for Graph Community Detection. In _2018 IEEE International Parallel and Distributed Processing Symposium, IPDPS 2018_. 885–895.
  * Goldberg and Harrelson (2005) Andrew V. Goldberg and Chris Harrelson. 2005\.  Computing the shortest path: _A_ search meets graph theory. In _SODA 2005_. 156–165.
  * Gonzalez et al. (2012) Joseph E. Gonzalez, Yucheng Low, and et al. 2012\.  PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs. In _OSDI 2012_. 17–30.
  * Gonzalez et al. (2014) Joseph E. Gonzalez, Reynold S. Xin, and et al. 2014\.  GraphX: Graph Processing in a Distributed Dataflow Framework. In _OSDI 2014_. 599–613.
  * Gropp (2001) William D Gropp. 2001.  Learning from the Success of MPI. In _Int. Conf. High-Perf. Comput._ 81–92.
  * Guo et al. (2017) Tao Guo, Xin Cao, and et al. 2017\.  Distributed Algorithms on Exact Personalized PageRank. In _SIGMOD 2017_. 479–494.
  * Gupta and Sundaram (2023) Shubhankar Gupta and Suresh Sundaram. 2023\.  Moving-Landmark Assisted Distributed Learning Based Decentralized Cooperative Localization (DL-DCL) with Fault Tolerance. In _AAAI 2023_. 6175–6182.
  * Han and Daudjee (2015) Minyang Han and Khuzaima Daudjee. 2015\.  Giraph Unchained: Barrierless Asynchronous Parallel Execution in Pregel-like Graph Processing Systems.  _Proc. VLDB Endow._ 8, 9 (2015), 950–961.
  * He and Wai (2021) Yiran He and Hoi-To Wai. 2021.  Provably Fast Asynchronous And Distributed Algorithms For Pagerank Centrality Computation. In _ICASSP 2021_. 5050–5054.
  * Heidari et al. (2018a) Safiollah Heidari, Yogesh Simmhan, and et al. 2018a.  Scalable graph processing frameworks: A taxonomy and open challenges.  _ACM Comput. Surv._ 51, 3 (2018), 1–53.
  * Heidari et al. (2018b) Safiollah Heidari, Yogesh Simmhan, and et al. 2018b.  Scalable Graph Processing Frameworks: A Taxonomy and Open Challenges.  _ACM Comput. Surv._ 51, 3 (2018), 60:1–60:53.
  * Henzinger, Krinninger and Nanongkai (2021) Monika Henzinger, Sebastian Krinninger, and et al. 2021\.  A Deterministic Almost-Tight Distributed Algorithm for Approximating Single-Source Shortest Paths.  _SIAM J. Comput._ 50, 3 (2021).
  * Hoang et al. (2019a) Loc Hoang, Vishwesh Jatala, and et al. 2019a.  DistTC: High Performance Distributed Triangle Counting. In _HPEC 2019_. 1–7.
  * Hoang et al. (2019b) Loc Hoang, Matteo Pontecorvi, and et al. 2019b.  A round-efficient distributed betweenness centrality algorithm. In _PPoPP 2019_. 272–286.
  * Holzer and Wattenhofer (2012) Stephan Holzer and Roger Wattenhofer. 2012\.  Optimal distributed all pairs shortest paths and applications. In _PODC 2012_. 355–364.
  * Hua et al. (2016) Qiang-Sheng Hua, Haoqiang Fan, and et al. 2016\.  Nearly Optimal Distributed Algorithm for Computing Betweenness Centrality. In _36th IEEE International Conference on Distributed Computing Systems, ICDCS 2016_. 271–280.
  * Jain et al. (2017) Chirag Jain, Patrick Flick, and et al. 2017\.  An Adaptive Parallel Algorithm for Computing Connected Components.  _IEEE Trans. Parallel Distributed Syst._ 28, 9 (2017), 2428–2439.
  * Jamour, Skiadopoulos and Kalnis (2018) Fuad T. Jamour, Spiros Skiadopoulos, and et al. 2018\.  Parallel Algorithm for Incremental Betweenness Centrality on Large Graphs.  _IEEE Trans. Parallel Distributed Syst._ 29, 3 (2018), 659–672.
  * Jeh and Widom (2002) Glen Jeh and Jennifer Widom. 2002\.  SimRank: a measure of structural-context similarity. In _KDD 2002_. 538–543.
  * Jiang (2016) Yichuan Jiang. 2016.  A Survey of Task Allocation and Load Balancing in Distributed Systems.  _IEEE Trans. Parallel Distributed Syst._ 27, 2 (2016), 585–599.
  * Kalavri, Vlassov and Haridi (2018) Vasiliki Kalavri, Vladimir Vlassov, and et al. 2018\.  High-Level Programming Abstractions for Distributed Graph Processing.  _IEEE Trans. Knowl. Data Eng._ 30, 2 (2018), 305–324.
  * Karypis and Kumar (1997) George Karypis and Vipin Kumar. 1997\.  METIS: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices.  (1997).
  * Kuhn (1955) Harold W Kuhn. 1955.  The Hungarian method for the assignment problem.  _Nav. Res. Logist. Q._ 2, 1-2 (1955), 83–97.
  * Lai et al. (2015) Longbin Lai, Lu Qin, and et al. 2015\.  Scalable Subgraph Enumeration in MapReduce.  8, 10 (2015), 974–985.
  * Lai et al. (2019) Longbin Lai, Zhu Qing, and et al. 2019\.  Distributed Subgraph Matching on Timely Dataflow.  _Proc. VLDB Endow._ 12, 10 (2019), 1099–1112.
  * Lai et al. (2023) Longbin Lai, Yufan Yang, and et al. 2023\.  GLogS: Interactive Graph Pattern Matching Query At Large Scale. In _USENIX ATC 23_. 53–69.
  * Lakhotia, Kannan and Prasanna (2018) Kartik Lakhotia, Rajgopal Kannan, and et al. 2018\.  Accelerating PageRank using Partition-Centric Processing. In _USENIX ATC 2018_. 427–440.
  * Lamm and Sanders (2022) Sebastian Lamm and Peter Sanders. 2022\.  Communication-efficient Massively Distributed Connected Components. In _IPDPS 2022_. 302–312.
  * Lenzen, Patt-Shamir and Peleg (2019) Christoph Lenzen, Boaz Patt-Shamir, and et al. 2019\.  Distributed Distance Computation and Routing with Small Messages.  _Distrib. Comput._ 32, 2 (2019), 133–157.
  * Li et al. (2015) Zhenguo Li, Yixiang Fang, and et al. 2015\.  Walking in the Cloud: Parallel SimRank at Scale.  _Proc. VLDB Endow._ 9, 1 (2015), 24–35.
  * Liao et al. (2022) Xuankun Liao, Qing Liu, and et al. 2022\.  Distributed D-core Decomposition over Large Directed Graphs.  _Proc. VLDB Endow._ 15, 8 (2022), 1546–1558.
  * Lin (2019) Wenqing Lin. 2019.  Distributed Algorithms for Fully Personalized PageRank on Large Graphs. In _WWW 2019_. 1084–1094.
  * Liu et al. (2023) Qing Liu, Xuankun Liao, and et al. 2023\.  Distributed ($\alpha$, $\beta$)-Core Decomposition over Bipartite Graphs. In _ICDE 2023_. 909–921.
  * Liu et al. (2019) Xu Liu, Jesun Sahariar Firoz, and et al. 2019\.  Distributed Direction-Optimizing Label Propagation for Community Detection. In _HPEC 2019_. 1–6.
  * Low et al. (2012) Yucheng Low, Joseph Gonzalez, and et al. 2012\.  Distributed GraphLab: A Framework for Machine Learning in the Cloud.  _Proc. VLDB Endow._ 5, 8 (2012), 716–727.
  * Lulli et al. (2017) Alessandro Lulli, Emanuele Carlini, and et al. 2017\.  Fast Connected Components Computation in Large Graphs by Vertex Pruning.  _IEEE Trans. Parallel Distributed Syst._ 28, 3 (2017), 760–773.
  * Luo (2019) Siqiang Luo. 2019.  Distributed PageRank Computation: An Improved Theoretical Study. In _AAAI 2019_. 4496–4503.
  * Luo (2020) Siqiang Luo. 2020.  Improved Communication Cost in Distributed PageRank Computation - A Theoretical Study. In _ICML 2020_ _(Proceedings of Machine Learning Research, Vol. 119)_. 6459–6467.
  * Luo, Wu and Kao (2022) Siqiang Luo, Xiaowei Wu, and et al. 2022\.  Distributed PageRank computation with improved round complexities.  _Inf. Sci._ 607 (2022), 109–125.
  * Luo and Zhu (2023) Siqiang Luo and Zulun Zhu. 2023.  Massively Parallel Single-Source SimRanks in o(log n) Rounds.  _CoRR_ abs/2304.04015 (2023).  arXiv:2304.04015
  * Luo et al. (2017) Xiongcai Luo, Jun Gao, and et al. 2017\.  UniWalk: Unidirectional Random Walk Based Scalable SimRank Computation over Large Graph. In _ICDE 2017_. 325–336.
  * Ma et al. (2018) Tinghuai Ma, Mingliang Yue, and et al. 2018\.  PSPLPA: Probability and similarity based parallel label propagation algorithm on spark.  _Physica A: Statistical Mechanics and its Applications_ 503 (2018), 366–378.
  * Maleki et al. (2016) Saeed Maleki, Donald Nguyen, and et al. 2016\.  DSMR: a shared and distributed memory algorithm for single-source shortest path problem. In _PPoPP 2016_. 39:1–39:2.
  * Malewicz et al. (2010) Grzegorz Malewicz, Matthew H. Austern, and et al. 2010\.  Pregel: a system for large-scale graph processing. In _SIGMOD 2010_. 135–146.
  * Malliaros et al. (2020) Fragkiskos D. Malliaros, Christos Giatsidis, and et al. 2020\.  The core decomposition of networks: theory, algorithms and applications.  _VLDB J._ 29, 1 (2020), 61–92.
  * Mandal and Hasan (2017) Aritra Mandal and Mohammad Al Hasan. 2017\.  A distributed k-core decomposition algorithm on spark. In _IEEE BigData 2017_. 976–981.
  * Mashreghi and King (2021) Ali Mashreghi and Valerie King. 2021\.  Broadcast and minimum spanning tree with o(m) messages in the asynchronous CONGEST model.  _Distributed Comput._ 34, 4 (2021), 283–299.
  * Modi et al. (2005) Pragnesh Jay Modi, Wei-Min Shen, and et al. 2005\.  ADOPT: Asynchronous distributed constraint optimization with quality guarantees.  _Artificial Intelligence_ 161, 1-2 (2005), 149–180.
  * Montresor, Pellegrini and Miorandi (2013) Alberto Montresor, Francesco De Pellegrini, and et al. 2013\.  Distributed k-Core Decomposition.  _IEEE Trans. Parallel Distributed Syst._ 24, 2 (2013), 288–300.
  * Moritz et al. (2018) Philipp Moritz, Robert Nishihara, and et al. 2018\.  Ray: A distributed framework for emerging AI applications. In _OSDI 2018_. 561–577.
  * Ngo et al. (2018) Hung Q. Ngo, Ely Porat, and et al. 2018\.  Worst-Case Optimal Join Algorithms.  _J. ACM_ 65, 3, Article 16 (2018).
  * Oracle (2014) Oracle. 2014.  Remote Method Invocation Home.  https://www.oracle.com/java/technologies/javase/remote-method-invocation-home.html
  * Pandey et al. (2021) Santosh Pandey, Zhibin Wang, and et al. 2021\.  Trust: Triangle Counting Reloaded on GPUs.  _IEEE Trans. Parallel Distributed Syst._ 32, 11 (2021), 2646–2660.
  * Pandurangan, Robinson and Scquizzato (2017) Gopal Pandurangan, Peter Robinson, and et al. 2017\.  A time- and message-optimal distributed algorithm for minimum spanning trees. In _STOC 2017_. 743–756.
  * Park, Myaeng and Kang (2016) Ha-Myung Park, Sung-Hyon Myaeng, and et al. 2016\.  PTE: Enumerating Trillion Triangles On Distributed Systems. In _KDD 2016_. 1115–1124.
  * Pearce, Gokhale and Amato (2014) Roger Pearce, Maya Gokhale, and et al. 2014\.  Faster parallel traversal of scale free graphs at extreme scale with vertex delegates. In _SC 2014_. IEEE, 549–559.
  * Peng et al. (2016) Yun Peng, Byron Choi, and et al. 2016\.  VColor: A practical vertex-cut based approach for coloring large graphs. In _ICDE 2016_. 97–108.
  * Peretz and Fischler (2022) Yossi Peretz and Yigal Fischler. 2022\.  A fast parallel max-flow algorithm.  _J. Parallel Distr. Com._ 169 (2022), 226–241.
  * Raghavan, Albert and Kumara (2007) Usha Nandini Raghavan, Réka Albert, and et al. 2007\.  Near linear time algorithm to detect community structures in large-scale networks.  _Physical review E_ 76, 3 (2007), 036106.
  * Reed et al. (2023) Emily A. Reed, Guilherme Ramos, and et al. 2023\.  A Scalable Distributed Dynamical Systems Approach to Learn the Strongly Connected Components and Diameter of Networks.  _IEEE Trans. Autom. Control._ 68, 5 (2023), 3099–3106.
  * Remis et al. (2018) Luis Remis, María Jesús Garzarán, and et al. 2018.  Exploiting social network graph characteristics for efficient BFS on heterogeneous chips.  _J. Parallel Distributed Comput._ 120 (2018), 282–294.
  * Roy, Mihailovic and Zwaenepoel (2013) Amitabha Roy, Ivo Mihailovic, and et al. 2013\.  X-Stream: edge-centric graph processing using streaming partitions. In _SOSP 2013_. 472–488.
  * Rozhon et al. (2022) Václav Rozhon, Christoph Grunau, and et al. 2022\.  Undirected (1+_$\epsilon$_)-shortest paths via minor-aggregates: near-optimal deterministic parallel and distributed algorithms. In _STOC 2022_. 478–487.
  * Sakr, Liu and Fayoumi (2013) Sherif Sakr, Anna Liu, and et al. 2013\.  The family of mapreduce and large-scale data processing systems.  _ACM Comput. Surv._ 46, 1 (2013), 11:1–11:44.
  * Salihoglu and Widom (2013) Semih Salihoglu and Jennifer Widom. 2013\.  GPS: a graph processing system. In _SSDBM 2013_. 22:1–22:12.
  * Sanders and Uhl (2023) Peter Sanders and Tim Niklas Uhl. 2023\.  Engineering a Distributed-Memory Triangle Counting Algorithm. In _IPDPS 2023_. 702–712.
  * Sariyüce et al. (2013) Ahmet Erdem Sariyüce, Erik Saule, and et al. 2013\.  STREAMER: A distributed framework for incremental closeness centrality computation. In _CLUSTER 2013_. 1–8.
  * Sarma et al. (2015) Atish Das Sarma, Anisur Rahaman Molla, and et al. 2015\.  Fast distributed PageRank computation.  _Theor. Comput. Sci._ 561 (2015), 113–121.
  * Sattar and Arifuzzaman (2018) Naw Safrin Sattar and Shaikh Arifuzzaman. 2018\.  Parallelizing Louvain Algorithm: Distributed Memory Challenges. In _DASC/PiCom/DataCom/CyberSciTech 2018_. 695–701.
  * Sattar and Arifuzzaman (2022) Naw Safrin Sattar and Shaikh Arifuzzaman. 2022\.  Scalable distributed Louvain algorithm for community detection in large graphs.  _J. Supercomput._ 78, 7 (2022), 10275–10309.
  * Shahrivari and Jalili (2021) Saeed Shahrivari and Saeed Jalili. 2021\.  Efficient Distributed k-Clique Mining for Large Networks Using MapReduce.  _IEEE Trans. Knowl. Data Eng._ 33, 3 (2021), 964–974.
  * Shao, Chen and Cui (2014) Yingxia Shao, Lei Chen, and et al. 2014\.  Efficient cohesive subgraphs detection in parallel. In _SIGMOD 2014_. 613–624.
  * Sharma, Seshadhri and Goel (2017) Aneesh Sharma, C Seshadhri, and et al. 2017\.  When hashes met wedges: A distributed algorithm for finding high similarity vectors. In _WWW 2017_. 431–440.
  * Shiraz et al. (2012) Muhammad Shiraz, Abdullah Gani, and et al. 2012\.  A review on distributed application processing frameworks in smart mobile devices for mobile cloud computing.  _IEEE Communications surveys & tutorials_ 15, 3 (2012), 1294–1313.
  * Simmhan et al. (2014) Yogesh Simmhan, Alok Gautam Kumbhare, and et al. 2014\.  GoFFish: A Sub-graph Centric Framework for Large-Scale Graph Analytics. In _Euro-Par 2014_ _(Lecture Notes in Computer Science, Vol. 8632)_. 451–462.
  * Smyth, Welling and Asuncion (2008) Padhraic Smyth, Max Welling, and et al. 2008\.  Asynchronous distributed learning of topic models.  _Advances in Neural Information Processing Systems_ 21 (2008).
  * Soudani, Fatemi and Nematbakhsh (2019) Nasrin Mazaheri Soudani, Afsaneh Fatemi, and et al. 2019\.  PPR-partitioning: a distributed graph partitioning algorithm based on the personalized PageRank vectors in vertex-centric systems.  _Knowl. Inf. Syst._ 61, 2 (2019), 847–871.
  * Steil et al. (2021) Trevor Steil, Tahsin Reza, and et al. 2021\.  TriPoll: computing surveys of triangles in massive-scale temporal graphs with metadata. In _SC 2021_. 67.
  * Strausz et al. (2022) András Strausz, Flavio Vella, and et al. 2022\.  Asynchronous Distributed-Memory Triangle Counting and LCC with RMA Caching. In _IPDPS 2022_. 291–301.
  * Sun et al. (2022) Changhao Sun, Huaxin Qiu, and et al. 2022\.  Better Approximation for Distributed Weighted Vertex Cover via Game-Theoretic Learning.  _IEEE Trans. Syst. Man Cybern. Syst._ 52, 8 (2022), 5308–5319.
  * Sun et al. (2020) Changhao Sun, Xiaochu Wang, and et al. 2020\.  Distributed Optimization for Weighted Vertex Cover via Heuristic Game Theoretic Learning. In _CDC 2020_. 325–330.
  * Svendsen, Mukherjee and Tirthapura (2015) Michael Svendsen, Arko Provo Mukherjee, and et al. 2015\.  Mining maximal cliques from a large graph using mapreduce: Tackling highly uneven subproblem sizes.  _Journal of Parallel and distributed computing_ 79 (2015), 104–114.
  * Talukder and Zaki (2016) Nilothpal Talukder and Mohammed J. Zaki. 2016\.  A distributed approach for graph mining in massive networks.  _Data Min. Knowl. Discov._ 30, 5 (2016), 1024–1052.
  * Teixeira et al. (2015) Carlos H. C. Teixeira, Alexandre J. Fonseca, and et al. 2015\.  Arabesque: a system for distributed graph mining. In _SOSP 2015_. 425–440.
  * The MPI Forum (1993) CORPORATE The MPI Forum. 1993.  MPI: a message passing interface. In _Proceedings of SC_. 878–883.
  * Tian et al. (2013) Yuanyuan Tian, Andrey Balmin, and et al. 2013\.  From ”Think Like a Vertex” to ”Think Like a Graph”.  _Proc. VLDB Endow._ 7, 3 (2013), 193–204.
  * Tomita, Tanaka and Takahashi (2004) Etsuji Tomita, Akira Tanaka, and et al. 2004\.  The Worst-Case Time Complexity for Generating All Maximal Cliques. In _COCOON 2004_ _(Lecture Notes in Computer Science, Vol. 3106)_. 161–170.
  * Toshniwal et al. (2014) Ankit Toshniwal, Siddarth Taneja, and et al. 2014\.  Storm@ twitter. In _Proceedings of SIGMOD_. 147–156.
  * Trinder, Loidl and Pointon (2002) Philip W. Trinder, H-W Loidl, and et al. 2002\.  Parallel and distributed Haskells.  _Journal of Functional Programming_ 12, 4-5 (2002), 469–510.
  * Wang et al. (2022) Yuanwei Wang, Huanqi Cao, and et al. 2022\.  Scaling Graph 500 SSSP to 140 Trillion Edges with over 40 Million Cores. In _SC 2022_. 19:1–19:15.
  * Wang et al. (2020) Yue Wang, Ruiqi Xu, and et al. 2020\.  DISK: A Distributed Framework for Single-Source SimRank with Accuracy Guarantee.  _Proc. VLDB Endow._ 14, 3 (2020), 351–363.
  * Wang et al. (2019) Zhaokang Wang, Rong Gu, and et al. 2019\.  BENU: Distributed Subgraph Enumeration with Backtracking-Based Framework. In _ICDE 2019_. 136–147.
  * Wehmuth and Ziviani (2013) Klaus Wehmuth and Artur Ziviani. 2013\.  DACCER: Distributed Assessment of the Closeness CEntrality Ranking in complex networks.  _Comput. Networks_ 57, 13 (2013), 2536–2548.
  * Weng et al. (2022) Tongfeng Weng, Xu Zhou, and et al. 2022\.  Efficient Distributed Approaches to Core Maintenance on Large Dynamic Graphs.  _IEEE Trans. Parallel Distributed Syst._ 33, 1 (2022), 129–143.
  * White (2012) Tom White. 2012.  _Hadoop: The definitive guide_.  ” O’Reilly Media, Inc.”.
  * Xiang et al. (2021) Lizhi Xiang, Arif Khan, and et al. 2021\.  cuTS: scaling subgraph isomorphism on distributed multi-GPU systems using trie based data structure. In _SC 2021_. 69.
  * Xu, Cheng and Fu (2016) Yanyan Xu, James Cheng, and et al. 2016\.  Distributed Maximal Clique Computation and Management.  _IEEE Trans. Serv. Comput._ 9, 1 (2016), 110–122.
  * Xu et al. (2014) Yanyan Xu, James Cheng, and et al. 2014\.  Distributed Maximal Clique Computation. In _IEEE BigData 2014_. 160–167.
  * Yan et al. (2014) Da Yan, James Cheng, and et al. 2014\.  Blogel: A Block-Centric Framework for Distributed Computation on Real-World Graphs.  _Proc. VLDB Endow._ 7, 14 (2014), 1981–1992.
  * Yan et al. (2015) Da Yan, James Cheng, and et al. 2015\.  Effective Techniques for Message Reduction and Load Balancing in Distributed Graph Computation. In _WWW 2015_. 1307–1317.
  * Yang et al. (2021) Zhengyi Yang, Longbin Lai, and et al. 2021\.  HUGE: An Efficient and Scalable Subgraph Enumeration System. In _SIGMOD 2021_. 2049–2062.
  * Yasar et al. (2022) Abdurrahman Yasar, Sivasankaran Rajamanickam, and et al. 2022\.  A Block-Based Triangle Counting Algorithm on Heterogeneous Environments.  _IEEE Trans. Parallel Distributed Syst._ 33, 2 (2022), 444–458.
  * Yu et al. (2020) Ziqiang Yu, Xiaohui Yu, and et al. 2020\.  Distributed Processing of k Shortest Path Queries over Dynamic Road Networks. In _SIGMOD 2020_. 665–679.
  * Zaharia et al. (2012) Matei Zaharia, Mosharaf Chowdhury, and et al. 2012\.  Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In _NSDI 2012_. 15–28.
  * Zaharia et al. (2010) Matei Zaharia, Mosharaf Chowdhury, and et al. 2010\.  Spark: Cluster Computing with Working Sets. In _HotCloud 2010_.
  * Zaharia et al. (2016) Matei Zaharia, Reynold S Xin, and et al. 2016\.  Apache spark: a unified engine for big data processing.  _Commun. ACM_ 59, 11 (2016), 56–65.
  * Zeng and Yu (2018) Jianping Zeng and Hongfeng Yu. 2018\.  A Scalable Distributed Louvain Algorithm for Large-Scale Graph Community Detection. In _CLUSTER 2018_. 268–278.
  * Zhang et al. (2017) Dongxiang Zhang, Dingyu Yang, and et al. 2017\.  Distributed shortest path query processing on dynamic road networks.  _VLDB J._ 26, 3 (2017), 399–419.
  * Zhou, Gao and Guan (2021) Tian Zhou, Lixin Gao, and et al. 2021\.  A Fault-Tolerant Distributed Framework for Asynchronous Iterative Computations.  _IEEE Trans. Parallel Distributed Syst._ 32, 8 (2021), 2062–2073.
  * Zhu et al. (2020) Huanzhou Zhu, Ligang He, and et al. 2020\.  WolfGraph: The edge-centric graph processing on GPU.  _Future Gener. Comput. Syst._ 111 (2020), 552–569.
  * Zuzic et al. (2022) Goran Zuzic, Gramoz Goranci, and et al. 2022\.  Universally-Optimal Distributed Shortest Paths and Transshipment via Graph-Based $\mathscr{l}$${}_{\mbox{1}}$-Oblivious Routing. In _SODA 2022_. 2549–2579.


# Breakdown of detailed balance for thermal radiation by synthetic fields

S.-A. Biehs<EMAIL_ADDRESS>Institut für Physik, Carl von
Ossietzky Universität, D-26111 Oldenburg, Germany  G. S. Agarwal
<EMAIL_ADDRESS>Institute for Quantum Science and Engineering and
Department of Biological and Agricultural Engineering Department of Physics
and Astronomy, Texas A & M University, College Station, Texas 77845, USA

###### Abstract

In recent times the possibility of non-reciprocity in heat transfer between
two bodies has been extensively studied. In particular the role of strong
magnetic fields has been investigated. A much simpler approach with
considerable flexibility would be to consider heat transfer in synthetic
electric and magnetic fields which are easily applied. We demonstrate the
breakdown of detailed balance for the heat transfer function
$\mathcal{T}(\omega)$, i.e. the spectrum of heat transfer between two objects
due to the presence of synthetic electric and magnetic fields. The spectral
measurements carry lot more physical information and were the reason for the
quantum theory of radiation. We demonstrate explicitly the synthetic field
induced non-reciprocity in the heat transfer transmission function between two
graphene flakes and for the Casimir coupling between two objects. Unlike many
other cases of heat transfer, the latter case has interesting features of the
strong coupling. Further the presence of synthetic fields affects the mean
occupation numbers of two membranes and propose this system for the
experimental verification of the breakdown of detailed balance.

Reciprocity and detailed balance are at the heart of Kirchhoff’s law stating
that the absorptivity equals emissivity for any frequency and angle of
incidence. In fact, the second law of thermodynamics enforces the reciprocity
or better detailed balance of the radiative heat transfer between two objects.
Here it is unimportant if far-field heat transfer where Planck’s blackbody
determines the upper limit is considered or near-field heat transfer where the
blackbody limit is not a limit anymore ReviewCarlos ; Limit ; RMP  as
experimentally tested by a great number of experiments HuEtAl2008 ;
OttensEtAl2011 ; Kralik2012 ; LimEtAl2015 ; WatjenEtAl2016 ; BernadiEtAl2016 ;
SongEtAl2016 ; FiorinoEtAl2018a ; FiorinoEtAl2018b  within the last decade.
How the second law enforces detailed balance can be understood Latella2017  by
considering the heat flux between two objects by first taking the transferred
power from object $a$ to object $b$

$P_{a\rightarrow b}=\int_{0}^{\infty}\frac{{\rm d}\omega}{2\pi}\hbar\omega
n_{a}(\omega)\mathcal{T}_{ab}(\omega)$ (1)

where $\hbar$ is the Planck constant, $n_{a}(\omega)=(\exp(\hbar\omega/k_{\rm
B}T_{a})-1)^{-1}$ is the photonic occupation number, $k_{\rm B}$ is the
Boltzmann constant, and $T_{a}$ is the temperature of object $a$. The quantity
$\mathcal{T}_{ab}(\omega)$ is a heat transfer function (HTF) for the heat flow
from object $a$ to object $b$. Similarly, the heat flow from object $b$ to
object $a$ is given by

$P_{b\rightarrow a}=\int_{0}^{\infty}\frac{{\rm d}\omega}{2\pi}\hbar\omega
n_{b}(\omega)\mathcal{T}_{ba}(\omega)$ (2)

with $n_{b}(\omega)=(\exp(\hbar\omega/k_{\rm B}T_{b})-1)^{-1}$ and $T_{b}$ the
temperatur of object $b$. In thermal equilibrium the objects have the same
temperature $T_{a}=T_{b}$ and therefore there is no net heat flow which means
that $P_{a\rightarrow b}=P_{b\rightarrow a}$ and hence

$\int_{0}^{\infty}\frac{{\rm d}\omega}{2\pi}\hbar\omega
n_{a}(\omega)\bigl{[}\mathcal{T}_{ba}(\omega)-\mathcal{T}_{ab}(\omega)\bigr{]}=0.$
(3)

Since this expression holds for any value of temperature $T_{a}=T_{b}$ and
therefore for different spectral weighting by $n_{a}$ it can be concluded that
the validity of the second law of thermodynamics is equivalent to the relation
$\mathcal{T}_{ab}(\omega)=\mathcal{T}_{ba}(\omega)$ regardless of any symmetry
FanRec . That means that even when time reversal symmetry is broken by
applying a magnetic field or using topological Weyl semi-metals, for instance,
detailed balance of the energy HTF must be fulfilled. However, in non-
reciprocal systems the detailed balance of thermal radiation can be nearly
completely violated when considering three objects ZhuEtAl2014  which also
offers applications for optimized non-reciprocal thermo-photovoltaic energy
conversion ParkEtAl2022 . Similarly, the HTFs do not need to fulfill
$\mathcal{T}_{ab}(\omega)=\mathcal{T}_{ba}(\omega)$ when at least a third
object $c$ or a non-reciprocal environment are present. In such many-body
systems therefore several interesting effects for thermal radiation in general
and radiative heat exchange in nanoparticle systems RMP  in particular could
be highlighted like persistent heat currents and heat fluxes zhufan ;
Silveirinha ; CircularHeatFlux , persistent spin and angular momenta
Silveirinha ; CircularHeatFlux ; Zubin2019 , giant thermal resistance
Latella2017 ; He2020 , a normal and anomalous Hall effect for thermal
radiation hall ; OttEtAl2019 ; ahall , as well as a diode effect with non-
reciprocal surface waves NRdiode . In all these studied systems, in order to
realize a non-reciprocal heat flux or a violation of detailed balance the
presence of a third body seems to be a necessary condition. However, within
the framework of fluctuational electrodynamics and the scattering formalism
Krueger1 ; Krueger2  a formal proof detailed in Ref. Herz  shows that
$\mathcal{T}_{ab}(\omega)=\mathcal{T}_{ba}(\omega)$ if the environment and the
objects fulfill both Lorentz reciprocity Alu . Therefore in principle for
radiative heat transfer between two objects with non-reciprocal properties in
a reciprocal environment detailed balance can be broken even though in
practive this has not been observed so far.

Interestingly, the presence of synthetic electric and magnetic fields offers
the possibility to break the detailed balance of energy transmission functions
even for only two coupled resonators which results in a non-reciprocal energy
transmission as shown theoretically and verified experimentally
PetersonEtAl2019 . The synthetic fields are generated by external modulation
of the resonance frequency of the two resonators which first of all generates
side bands which can be understood by the presence of a synthetic electric
field in the synthetic frequency domain YuanEtAl2018 . When the modulation of
the two resonators is phase-shifted a synthetic magnetic field for the photons
is generated TzuanEtAl2014  which enables the Aharonov-Bohm effect for photons
FangEtAl2012 , for instance. Now, dynamic modulations of temperatures or
material properties have also been considered for modulation of radiative heat
exchange between two or more objects ZwolVo2 ; Ito ; Minnich ; Minnich2
showing that the modulation of the temperature or chemical potential can
result in a shuttling effect Latella  and the modulation of material
properties can be used to modulate the radiative heat flux between two or more
objects Cuevas ; Messina . However, in all those works the HTF for the
radiative heat exchange between two bodies are again strictly fulfilling
detailed balance, i.e. $\mathcal{T}_{ab}(\omega)=\mathcal{T}_{ba}(\omega)$.

In this letter, by using a quantum Langevin equation approach to treat heat
transfer we show that synthetic fields can lead to a breakdown of detailed
balance for the HTF between two resonant objects, i.e. we explicitely show
that $\mathcal{T}_{ab}(\omega)\neq\mathcal{T}_{ba}(\omega)$. We further show
that this broken detailed balance does not result in a non-reciprocal heat
flux, i.e. we still have $P_{a\rightarrow b}=P_{b\rightarrow a}$ and the
validity of Eq. (3). We will discuss these features for the radiative heat
flux between two graphene flakes in which case the synthetic fields are
realized by modulating Fermi energies. Furthermore, we propose to measure the
broken detailed balance in the strong-coupling regime of two Casimir-force
coupled membranes as used in recent experiments like in Ref. FongEtAl .

In the following we describe the near-field radiative heat flux between two
graphene flakes as well as Casimir force coupled membranes by two coupled
oscillators JOSA ; Barton ; Karthik . The oscillator frequencies
$\omega_{a/b}$ then correspond to the frequencies of the main optical or
vibrational modes of the graphene flakes or the membranes and their damping is
described by the damping constants $\kappa_{a/b}$. The coupling strength
between the oscillators $g$ quantifies the interaction strength of the
graphene flakes or membranes due to the fluctuational electromagnetic fields
which are at the origin of the radiative heat transfer and Casimir force. Then
the coupled oscillators can be described by a set of two quantum Langevin
equations AgarwalBook2012 ; Suppl

$\displaystyle\dot{a}$ $\displaystyle=-{\rm i}\omega_{a}a-\kappa_{a}a-{\rm
i}gb+F_{a},$ (4) $\displaystyle\dot{b}$ $\displaystyle=-{\rm
i}\omega_{b}b-\kappa_{b}b-{\rm i}ga+F_{b}$ (5)

for the lowering operators $a$ and $b$ of the two coupled oscillators.
Furthermore, both oscillators are assumed to be coupled to their own baths
which enter here through the bath operators $F_{a/b}$ into the description.

Figure 1:  Sketch of the forward heat flux $P_{a\rightarrow b}$ in the
considered two couples oscillators with periodic modulation in the synthetic
dimension with the synthetic electric and magnetic fields $\mathbf{E}$ and
$\mathbf{B}$.

Now, we introduce synthetic electric and magnetic fields via the identical
frequency modulation of both oscillators

$\omega_{a}\rightarrow\omega_{a}+\beta\cos(\Omega
t)\quad\text{and}\quad\omega_{b}\rightarrow\omega_{b}+\beta\cos(\Omega
t+\theta)$ (6)

with modulation frequency $\Omega$, amplitude $\beta$ and with a phase shift
$\theta$. By Fourier transforming the coupled Langevin equations into
frequency space we obtain the set of equations in the compact form

$\boldsymbol{\psi}=\mathds{M}\mathbf{F}+\frac{\beta}{2{\rm
i}}\mathds{M}\mathds{Q}_{+}\boldsymbol{\psi}_{+}+\frac{\beta}{2{\rm
i}}\mathds{M}\mathds{Q}_{-}\boldsymbol{\psi}_{-}$ (7)

by introducing the vectors
$\boldsymbol{\psi}=\bigl{(}a(\omega),b(\omega)\bigr{)}^{t}$,
$\boldsymbol{\psi}_{\pm}=\bigl{(}a(\omega\pm\Omega),b(\omega\pm\Omega)\bigr{)}^{t}$,
and $\mathbf{F}=\bigl{(}F_{a}(\omega),F_{b}(\omega)\bigr{)}^{t}$, and the
matrices

$\mathds{M}=\mathds{A}^{-1}\quad\text{with}\quad\mathds{A}=\begin{pmatrix}X_{a}&{\rm
i}g\\\ {\rm i}g&X_{b}\end{pmatrix}$ (8)

so that

$\mathds{M}=\frac{1}{X_{a}X_{b}+g^{2}}\begin{pmatrix}X_{b}&-{\rm i}g\\\ -{\rm
i}g&X_{a}\end{pmatrix}$ (9)

introducing $X_{a/b}=-{\rm i}(\omega-\omega_{a/b})+\kappa_{a/b}$ and the
diagonal matrix $\mathds{Q}_{\pm}={\rm diag}\bigl{(}1,{\rm e}^{\pm{\rm
i}\theta}\bigr{)}$. This compact form makes obvious that we have an infinite
set of equations in frequency space due to the coupling to the sidebands
$\pm\Omega$, $\pm 2\Omega$, etc. introduced by the modulation. These sidebands
can be understood as generated by an electric synthetic field along the
synthetic frequency axis (see Fig. 1). Furthermore, the phase shift itself can
be interpreted by a synthetic magnetic field PetersonEtAl2019  which adds a
phase $\mathds{Q}_{+}$ for “upward” and $\mathds{Q}_{-}$ for “downward”
transitions in the frequency bands. Recently, it has been shown theoretically
and experimentally that this synthetic magnetic field results in non-
reciprocal energy transmission in coupled oscillator systems PetersonEtAl2019
. From the mathematical expression of $\mathds{Q}_{\pm}$ it is clear that
$\mathds{Q}_{+}=\mathds{Q}_{-}$ for phases $\phi=l\pi$ for all integers $l$.
Hence for such phases the synthetic magnetic field makes no difference for
“upward” and “downward” transitions and we can expect that there is no
breaking of detailed balance by the synthetic magnetic field.

Furthermore, the compact matrix form allows us to write down formally the
infinite set of equations in frequency space. To this end, we introduce the
infinitely large block vectors

$\displaystyle\uline{\boldsymbol{\psi}}$
$\displaystyle=(\ldots,\boldsymbol{\psi}_{++},\boldsymbol{\psi}_{+},\boldsymbol{\psi},\boldsymbol{\psi}_{-},\boldsymbol{\psi}_{--},\ldots)^{t}$
(10) $\displaystyle\uline{\mathbf{F}}$
$\displaystyle=(\ldots,\mathbf{F}_{++},\mathbf{F}_{+},\mathbf{F},\mathbf{F}_{-},\mathbf{F}_{--},\ldots)^{t}$
(11)

where the indices are defined as
$\mathbf{F}_{\pm}=\bigl{(}F_{a}(\omega\pm\Omega),F_{b}(\omega\pm\Omega)\bigr{)}^{t}$,
$\mathbf{F}_{++/--}=\bigl{(}F_{a}(\omega\pm 2\Omega),F_{b}(\omega\pm
2\Omega)\bigr{)}^{t}$, etc. Then we can rewrite the coupled Langevin Eqs. (7)
as

$\uline{\boldsymbol{\psi}}=\uline{\mathds{L}}^{-1}\uline{\mathds{M}}\uline{\mathbf{F}}$
(12)

where the diagonal and tridiagonal block matrices $\uline{\mathds{M}}$ and
$\uline{\mathds{L}}$ are defined in Ref. Suppl . For any solution of this
matrix equation it is necessary to consider only a finite subset. As typically
done in such a Floquet-Shirley type approach we consider only block vectors of
size $2(2N+1)$ with the corresponding block matrices of size $2(2N+1)\times
2(2N+1)$ centered around the solution for the zeroth sideband. The result can
be considered as a perturbation result up to order $N$.

Finally, we can derive a general expression for the spectral correlation
functions $\langle a^{\dagger}a\rangle_{\omega}$, $\langle
b^{\dagger}b\rangle_{\omega}$, $\langle a^{\dagger}b\rangle_{\omega}$, and
$\langle b^{\dagger}a\rangle_{\omega}$. To this end, we first separate the
contributions to $\uline{\boldsymbol{\psi}}$ due to the bath operator
$\uline{\mathbf{F}}_{a}$ and $\uline{\mathbf{F}}_{b}$ by introduce the two
block matrices $\uline{\mathds{Y}}_{a}={\rm diag}(1,0,1,0,\ldots)$ and
$\uline{\mathds{Y}}_{b}={\rm diag}(0,1,0,1,\ldots)$ so that
$\uline{\mathds{Y}}_{a}+\uline{\mathds{Y}}_{b}=\uline{\mathds{1}}$. These two
matrices allow us to split the contributions from bath $a$ and bath $b$ so
that

$\uline{\boldsymbol{\psi}}=\uline{\mathds{L}}^{-1}\uline{\mathds{M}}\mathds{Y}_{a}\uline{\mathbf{F}}+\uline{\mathds{L}}^{-1}\uline{\mathds{M}}\mathds{Y}_{b}\uline{\mathbf{F}}.$
(13)

We assume that the bath operators fulfill the fluctuation-dissipation theorem
in the form ($i,j=a,b$)

$\langle
F_{i}^{\dagger}(\omega+l\Omega)F_{j}(\omega^{\prime}+l^{\prime}\Omega)\rangle=\delta_{i,j}\delta_{l,l^{\prime}}2\pi\delta(\omega-\omega^{\prime})\langle
F_{i}^{\dagger}F_{i}\rangle_{\omega},$ (14)

with $\langle
F_{a}^{\dagger}F_{a}\rangle_{\omega}=2\kappa_{a}n_{a}(\omega_{a})$ and
$\langle F_{b}^{\dagger}F_{b}\rangle_{\omega}=2\kappa_{b}n_{b}(\omega_{b})$.
This means that we assume that both baths are uncoupled and that the bath
spectra are white noise spectra. This is a good assumptions as long as all
important sidebands are close to $\omega_{a/b}$. This assumption can be made
as long as the modulation amplitudes and the modulation frequency are small
enough. Therewith we arrive at the final result

$\langle\uline{\boldsymbol{\psi}}_{\alpha}^{\dagger}\uline{\boldsymbol{\psi}}_{\beta}\rangle_{\omega}=\sum_{j=a,b}2\kappa_{j}n_{j}(\omega_{j})\bigl{(}\uline{\mathds{L}}^{-1}\uline{\mathds{M}}\uline{\mathds{Y}}_{j}\uline{\mathds{M}}^{\dagger}{\uline{\mathds{L}}^{-1}}^{\dagger}\bigr{)}_{\beta,\alpha}$
(15)

using the properties
$\uline{\mathds{Y}}_{a/b}^{\dagger}=\uline{\mathds{Y}}_{a/b}$ and
$\uline{\mathds{Y}}_{a/b}\uline{\mathds{Y}}_{a/b}=\uline{\mathds{Y}}_{a/b}$.
From this expression we can numerically calculate all spectral correlation
functions. For instance $\langle a^{\dagger}a\rangle_{\omega}$ is given by the
component $\alpha=2N+1$ and $\beta=2N+1$, $\langle
a^{\dagger}b\rangle_{\omega}$ by the component $\alpha=2N+1$ and $\beta=2N+2$,
etc. Note, that the such obtained spectral correlation functions are the sum
of all sideband frequency components with equal weighting factors
$2\kappa_{a}n_{a}$ and $2\kappa_{b}n_{b}$ .

Figure 2:  Non-reciprocal HTF $\mathcal{T}_{ab/ba}$ for graphene flakes at
distance $d=100\,{\rm nm}$ using perturbation order $N=20$. Top:
$\mathcal{T}_{ab}$ for $n_{a}\neq 0$ and $n_{b}=0$. Middle: $\mathcal{T}_{ba}$
for $n_{b}\neq 0$ and $n_{a}=0$. Parameters: $\beta=0.05\omega_{p}$,
$\Omega=0.05\omega_{p}$, and $\theta$ is varied. Bottom: $\mathcal{T}_{ab}$
(full lines) and $\mathcal{T}_{ba}$ (dashed lines) for
$\theta=0.3\pi,0.5\pi,0.7\pi$ with zero lines shifted to 3, 6, 9, resp.

Let us now use the model to discuss the heat flux between two graphene flakes.
Graphene flakes have sharp resonances like plasmonic nanoparticles. The
permittivity of a graphene flake lying within a plane parallel to the x-y
plane is given by the polarizability tensor $\uuline{\alpha}={\rm
diag}(\alpha,\alpha,0)$ with Graphene

$\alpha=\frac{3c^{3}k_{r}}{2\omega_{p}^{2}}\frac{1}{\omega_{p}^{2}-\omega^{2}-{\rm
i}k\omega}$ (16)

with plasma frequency $\omega_{p}$, amplitude $k_{r}$, and damping constant
$k$ which depent on the Fermi energies $E_{F}$ (in eV) Suppl  and can be
changed electrically or optically so that a modulation of the resonance
frequency is feasible Zwol ; Minnich2 . The HTF between two identical graphene
flakes facing each other at a distance $d$ is within fluctuational
electrodynamics in the quasi-static regime given by Suppl

$\mathcal{T}_{ab}(\omega)=\mathcal{T}_{ba}(\omega)=8\frac{(\alpha^{\prime\prime})^{2}}{(4\pi
d^{3})^{2}}\frac{1}{|1+\frac{\alpha^{2}}{(4\pi d^{3})^{2}}|^{2}}.$ (17)

This HTF can now be related to our model. Within our model, by setting
$F_{b}=0$ the steady state power from $a$ to $b$ is Suppl

$P_{a\rightarrow b}=\int_{0}^{\infty}\frac{{\rm
d}\omega}{2\pi}\hbar\omega_{a}2\kappa_{a}\langle b^{\dagger}b\rangle_{\omega}$
(18)

so that the HTF is

$\mathcal{T}_{ab}=\frac{2\kappa_{a}}{n_{a}(\omega_{a})}\langle
b^{\dagger}b\rangle_{\omega}.$ (19)

Similarly, $\mathcal{T}_{ba}$ can be obtained by exchanging $a$ and $b$.
Without any modulation we can directly determine the HTF from Eq. (7) for the
case $\beta=0$ and taking $F_{b}=0$. Then we obtain Suppl

$\mathcal{T}_{ab}=\frac{4(g\kappa_{a})^{2}}{|X_{a}^{2}+g^{2}|^{2}}.$ (20)

By identifying the resonance frequency $\omega_{a}=\omega_{b}$ with the plasma
frequency $\omega_{p}$ and $\kappa_{a}=\kappa_{b}\equiv\kappa$ with damping
constant $k$ of the graphene sheet we can fit the HTF of our model to that of
Eq. (17). We obtain a very good spectral fit for $g=0.011\kappa_{a}$ (see
Supplemental Material Suppl ).

Figure 3:  Top: Total heat flux $P_{a\rightarrow b}$ between two graphene
flakes at distance $d=100\,{\rm nm}$ ($g/\kappa=3.9$) for $T_{a}=300\,{\rm K}$
and $T_{b}=0\,{\rm K}$ as function of the modulation amplitude $\beta$ using
dephasings $\theta=0.1\pi,0.3\pi,0.5\pi,1.0\pi$. Bottom:
$P_{0}/P_{a\rightarrow b}$ as function of $\beta$ and $\Omega$ where $P_{0}$
is the value without modulation for $\theta=1.0\pi$. Numerical calculation is
done for $N=20$.

In Fig. 2 we show the numerical results for the HTF for two identical graphene
sheets with $\omega_{a}=\omega_{b}=\omega_{p}=1.69\times 10^{14}\,{\rm rad/s}$
and $\kappa_{a}=\kappa_{b}=0.013\omega_{p}$ for $E_{F}=0.4\,{\rm eV}$ when the
resonance frequencies are modulated as in Eqs. (6). We choose a relatively
small amplitude $\beta=0.05\omega_{p}$ which approximately corresponds to a
change of the Fermi energy by $0.05\,{\rm eV}$ and a relatively small
modulation frequency $\Omega=0.05\omega_{p}$. First of all it can be seen that
as expected the modulation produces side bands around the resonance frequency
$\omega_{p}$. More interesting is that the spectra are in general different
for $\theta\neq l\pi$ ($l\in\mathbf{Z}$) so that the detailed balance between
the HTF is broken and we clearly have
$\mathcal{T}_{ab}(\omega)\neq\mathcal{T}_{ba}(\omega)$ which is clearly due to
the synthetic electric and magnetic fields. However, the integrated heat flux
shown in Fig. 3 is reciprocal so that we find $P_{a\rightarrow
b}=P_{b\rightarrow a}$ which is equivalent to saying that the integrals over
$\mathcal{T}_{ab}$ and $\mathcal{T}_{ba}$ are exactly the same which is in
full agreement with the general statement in Eq. (3). Another feature is that
the heat flux can be inhibited due to the modulation which can be easily
understood by the fact that the resonances do less overlap during a modulation
cycle when they are phase shifted. It is an interesting feature that this
inhibition can be extremely high for specific combinations of $\Omega$ and
$\beta$ in particular for $\beta\approx\Omega$ the heat flux can be up to 300
times smaller than without modulation for moderate values of modulation
frequency and amplitude.

Next, we consider a system of two membranes coupled by Casimir forces which
allow for measurements of the spectra of the mean occupation numbers $\langle
a^{\dagger}a\rangle_{\omega}$ and $\langle b^{\dagger}b\rangle_{\omega}$ of
the membranes as done in Ref. FongEtAl , for instance. In that work the
parameters are given by $\kappa_{a}=\omega_{a}/(2Q_{a})$ and
$\kappa_{b}=\omega_{b}/(2Q_{b})$ for the damping with oscillation frequencies
$\omega_{a}=\omega_{b}=2\pi\cdot 191.6\,{\rm kHz}\equiv\omega_{0}$ and quality
factors $Q_{a}=4.5\times 10^{4}$ and $Q_{b}=2\cdot 10^{4}$. For an unambiguous
identification of the impact of the synthetic fields we choose
$\kappa_{a}=\kappa_{b}=10\omega_{a}/(2Q_{a})\equiv\kappa$ which is much larger
than in the actual experiment. The coupling constant due to the Casimir force
is $g(d)=d^{-4.91}2\times 10^{-30}\,s^{-1}$. The measurements were carried out
for distances from $d=300\,{\rm nm}$ which is in the strong coupling regime
($g/\kappa=1.54$) to $d=800\,{\rm nm}$ in the weak coupling regime
($g/\kappa_{1}=0.013$). The transition between both regimes ($g/\kappa=1$)
occurs at a distance of about 330nm .

In Fig. 4 we show the HTFs $\mathcal{T}_{ab}$ and $\mathcal{T}_{ba}$ in the
strong-coupling regime. The broken detailed balance in the strong-coupling can
be nicely seen. For an experimental verification a measurement of the mean
occupation numbers as done in Ref. FongEtAl  can be made which show imbalances
directly connected with the broken detailed balance (see Fig. 3 and 4 in Suppl
). However, when assuming that $n_{a}=n_{b}$ we find that $\langle
a^{\dagger}a\rangle/n_{a}+\langle a^{\dagger}a\rangle/n_{b}$ equals exactly
$\langle b^{\dagger}b\rangle/n_{a}+\langle b^{\dagger}b\rangle/n_{b}$. Hence
in global equilibrium the synthetic field has no impact on the total
occupation numbers of the membranes which coincides with the result that the
heat flux is reciprocal, i.e. $P_{a\rightarrow b}=P_{b\rightarrow a}$.
Therefore the impact of the synthetic fields can only be measured when both
membranes have different temperatures as realized in Ref. FongEtAl . It has to
be emphasized that the broken detailed balance due to the synthetic fields as
seen in the imbalance of the occupation numbers becomes prominent in strong-
coupling regime Suppl .

Figure 4:  HTFs $\mathcal{T}_{ab}$ (full lines) and $\mathcal{T}_{ba}$ (dashed
lines) of two coupled membranes in strong coupling regime with $d=300\,{\rm
nm}$ with modulation parameters $\Omega=0.0005\omega_{0}$,
$\beta=0.0005\omega_{0}$, and $\theta=\pi/2$. Numerical calculation is done
for $N=10$.

In conclusion, we have shown that the presence of electric and magnetic
synthetic fields breaks the detailed balance of the HTF but without resulting
into a non-reciprocal total heat flux between two objects. We have discussed
this phenomenon for the near-field radiative heat transfer between two
graphene flakes. Furthermore we could show that synthetic fields allow for a
strong heat flux inhibition which can be used to thermally isolate the
graphene flakes by periodic modulations. Finally, we propose to measure the
breakdown of detailed balance by measuring the mean occupation numbers of
Casimir forced coupled membranes having different temperatures as recently
done without dynamic modulation.

###### Acknowledgements.

S.-A. B. acknowledges helpful discussions with Philippe Ben-Abdallah and
support from Heisenberg Programme of the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) under the project No. 404073166. This research was
supported in part by the National Science Foundation under Grant No. NSF
PHY-1748958. G.S. A. thanks the kind support of The Air Force Office of
Scientific Research [AFOSR award no. FA9550-20-1-0366], The Robert A. Welch
Foundation [grant no. A-1943] and the Infosys Foundation Chair of the
Department of Physics, IISc Bangalore.

## References

  * (1) J. C. Cuevas and F. J. Garcia-Vidal, ACS Photonics 5, 3896 (2018).
  * (2) P. S. Venkataram, S. Molesky, W. Jin, and A. W.Rodriguez, Phys. Rev. Lett. 124, 013904 (2020)
  * (3) S.-A. Biehs, R. Messina, P. S. Venkataram, A. W. Rodriguez, J. C. Cuevas, and P. Ben-Abdallah, Rev. Mod. Phys. 93, 025009 (2021).
  * (4) L. Hu, A. Narayanaswamy, X. Chen, G. Chen, Appl. Phys. Lett. 92, 133106 (2008).
  * (5) R. S. Ottens, V. Quetschke, S. Wise, A. A. Alemi, R. Lundock, G. Mueller, D. H. Reitze, D. B. Tanner, B. F. Whiting, Phys. Rev. Lett. 107, 014301 (2011).
  * (6) T. Kralik, P. Hanzelka, M. Zobac, V. Musilova, T. Fort, M. Horak, Phys. Rev. Lett. 109, 224302 (2012).
  * (7) M. Lim, S. S. Lee, B. J. Lee, Phys. Rev. B 91, 195136 (2015).
  * (8) J. I. Watjen, B. Zhao, Z. M. Zhang, Appl. Phys. Lett. 109, 203112 (2016).
  * (9) M. P. Bernardi, D. Milovich. M. Francoeur, Nature Comm. 7, 12900 (2016).
  * (10) B. Song, D. Thompson, A. Fiorino, Y. Ganjeh, P. Reddy, E. Meyhofer, Nature Nanotechn. 11, 509 (2016).
  * (11) A. Fiorino, D. Thompson, L. Zhu, R. Mittapally, S.-A. Biehs, O. Bezencenet, N. El-Bondry, S. Bansropun, P. Ben-Abdallah, E. Meyhofer, P. Reddy, ACS Nano 12, 5774 (2018).
  * (12) A. Fiorino, D. Thompson, L. Zhu, B. Song, P. Reddy, E. Meyhofer, Nano Letters 18, 3711 (2018).
  * (13) I. Latella and P. Ben-Abdallah, Phys. Rev. Lett. 118, 173902, (2017).
  * (14) C. Guo and S. Fan, Phys. Rev. B 102, 085401 (2020).
  * (15) L. Zhu, S. Fan, Phys. Rev. B 90, 220301(R) (2014).
  * (16) Y. B. Park, B. Zhao, and S. H. Fan, Nano Lett. 22, 448 (2022).
  * (17) L. Zhu and S. Fan, Phys. Rev. Lett. 117, 134303 (2016).
  * (18) M. G. Silveirinha, Phys. Rev. B 95, 115103 (2017).
  * (19) A. Ott, P. Ben-Abdallah, and S.-A. Biehs, Phys. Rev. B 97, 205414 (2018).
  * (20) C. Khandekar, Z. Jacob, New J. Phys. 21, 103030 (2019).
  * (21) M.-J. He, H. Qi, Y.-X Su, Y.-T. Ren, Y.-J. Zhao, and M. Antezza, Appl. Phys. Lett. 117, 113104 (2020).
  * (22) P. Ben-Abdallah, Phys. Rev. Lett. 116, 084301, (2016).
  * (23) A. Ott, R. Messina, P. Ben-Abdallah, and S.-A. Biehs, J. Photon. Energy 9, 032711 (2019).
  * (24) A. Ott, S.-A. Biehs, P. Ben-Abdallah, Phys. Rev. B 101, 241411(R) (2020).
  * (25) A. Ott, R. Messina, P. Ben-Abdallah and S.-A. Biehs, Appl. Phys. Lett. 114, 163105 (2019).
  * (26) M. Krüger, T. Emig, and M. Kardar, Phys. Rev. Lett. 106, 210404 (2011).
  * (27) M. Krüger, G. Bimonte, T. Emig, and M. Kardar, Phys. Rev. B 86, 115423 (2012).
  * (28) F. Herz and S.-A. Biehs, EPL 127, 44001 (2019).
  * (29) C. Caloz, A. Alu, S. Tretyakov, D. Sounas, K. Achouri, and Z.-L. Deck-Leger, Phys. Rev. Appl. 10, 047001 (2018).
  * (30) C. W. Peterson, W. A. Benalcazar, M. Lin, T. L. Hughes, and G. Bahl, Phys. Rev. Lett. 123, 063901 (2019).
  * (31) L. Yuan, Q. Lin, M. Xiao, and S. Fan, Optica 5, 1396 (2018).
  * (32) L. D. Tzuang, K. Fang, P. Nussenzveig, S. Fan, and M. Lipson, Nat. Photonics 8, 701 (2014).
  * (33) K. Fang, Z. Yu, and S. Fan, Phys. Rev. Lett. 108, 153901 (2012).
  * (34) P. J. van Zwol, L. Ranno, and J. Chevrier, Phys. Rev. Lett. 108, 234301 (2012).
  * (35) K. Ito, K. Nishikawa, A. Miura, H. Toshiyoshi, and H. Iizuka, Nano Lett. 17, 4347 (2017).
  * (36) J. Kou and A. J. Minnich, Opt. Exp. 26, A730 (2018).
  * (37) N. H. Thomas, M. C. Sherrott, J. Broulliet, H. A. Atwater, and A. J. Minnich, Nano Lett. 19, 3898 (2019).
  * (38) I. Latella, R. Messina, J. M. Rubi, and P. Ben-Abdallah, Phys. Rev. Lett. 121, 023903 (2018).
  * (39) R. M. Abraham Ekeroth, P. Ben-Abdallah, J.C. Cuevas, and A. Garcia Martin, ACS Photonics 5, 705 (2017).
  * (40) R. Messina, A. Ott, C. Kathmann, S.-A. Biehs, and P. Ben-Abdallah, Phys. Rev. B 103, 115440 (2021).
  * (41) K. Y. Fong, H.-K. Li, R. Zhao, S. Yang, Y. Wang, and X. Zhang, Nature 576, 243 (2020).
  * (42) S.-A. Biehs and G. S. Agarwal, J. Opt. Soc. Am. B 30, 700 (2013).
  * (43) G. Barton, J. Stat. Phys. 165, 1153 (2016).
  * (44) K. Sasihithlu and G.S. Agarwal, Nanophotonics 7, 1581(2018).
  * (45) G. S. Agarwal, Quantum Optics, (Cambridge Univ. Press, Cambridge, 2012) .
  * (46) See Supplemental Material at …
  * (47) S. Thongrattanasiri, F. H.L. Koppens, F. J. G. de Abajo, Phys. Rev. Lett. 108, 047401 (2012).
  * (48) P. J. van Zwol and S. Thiele and C. Berger and W. A. de Heer and J. Chevrier, Phys. Rev. Lett. 109, 264301 (2012).


# Context Propagation from Proposals for Semantic Video Object Segmentation

###### Abstract

In this paper, we propose a novel approach to learning semantic contextual
relationships in videos for semantic object segmentation. Our algorithm
derives the semantic contexts from video object proposals which encode the key
evolution of objects and the relationship among objects over the spatio-
temporal domain. This semantic contexts are propagated across the video to
estimate the pairwise contexts between all pairs of local superpixels which
are integrated into a conditional random field in the form of pairwise
potentials and infers the per-superpixel semantic labels. The experiments
demonstrate that our contexts learning and propagation model effectively
improves the robustness of resolving visual ambiguities in semantic video
object segmentation compared with the state-of-the-art methods.

Index Terms—  Semantic context, semantic video object segmentation

##  1 Introduction

Semantic video object segmentation, which aims to assign a semantic label to
every pixel in video frames, is an essential step in various computer vision
and multimedia analysis tasks. Recent years have witnessed significant
attention and progress toward this problem. However, in addition to fast
motion, appearance variations, pose change, and occlusions, the difficulty in
resolving the inherent semantic ambiguities still plagues the robustness and
accuracy of such a large scale semantic labeling problem.

Recently, weakly supervised methods utilizing robust detection and tracking
approach have been proposed to address the semantic ambiguity issue. In the
learning process, object-relevant instances _i.e_. object proposals sharing
the same semantic tags are usually selected from video frames to learn
category or instance specific models for semantic labeling. Early work trained
classifiers to incorporate scene topology and semantics into pixel-level
object detection and localization [1]. Both object detector and tracker were
employed to either impose spatio-temporal coherence [2, 3] or learn an
appearance model [4] for encoding the appearance variation of semantic
objects. Lately, video object proposal selection algorithm has also been
proposed to integrate longer-range object reasoning with superpixel labeling
[5]. Despite of significant advances that have been made by the above methods,
global contextual relationships between semantic video objects remain under-
explored. Yet, contextual relationships are ubiquitous and provide important
cues for scene understanding related tasks.

Pairwise contextual relationships have been investigated in image segmentation
[6] and object detection [7, 8] tasks, whose importance has been highlighted
in effectively resolving semantic ambiguities. However, these methods model
the contextual relationships in terms of co-occurrence of higher-level
statistics of object categories, which favors frequently appeared classes in
the training data to enforce rigid semantic labelling. Small objects are more
likely to be omitted due to the sensitivity to the number of pixels that
objects occupy.

Graphical models have become powerful tools in computer vision [9, 10, 11, 12,
13, 14], providing a versatile framework for modeling contextual
relationships. These approaches exploit the inherent structure and
relationships within images [15, 16, 17] and videos [18, 19, 20], facilitating
more sophisticated and context-aware analysis. By representing visual elements
as nodes and their interactions as edges, graphical models capture spatial,
temporal, and semantic dependencies essential for tasks such as visual
information retrieval [21], stylization [22, 23, 24], object detection [25,
26, 27], scene understanding [28, 29, 30, 31, 32], and image or video
segmentation [33, 4, 34, 35, 36, 37, 38, 39, 40, 41].

In this work, we propose a novel model to exploit and propagate contextual
relationships among video object proposals without relying on training data.
Such a way of modeling spatio-temporal object contextual relationships has not
been well studied. Our model is able to capture the intra- and inter- category
contextual relationships by considering the content of an input video in a
non-parametric approach. This context model is comprised of a set of spatial-
temporal context exemplars which provide a novel interpretation of contextual
relationships in a link view which formulates the problem of learning
contextual relationships as the label propagation problem on a similarity
graph. This similarity graph naturally reflects the intrinsic and extrinsic
relationship between semantic objects in the spatial-temporal domain. Due to
the sparsity of this similarity graph, the learning process can be very
efficient.

##  2 Our Approach

In this section, we describe our proposed context model and how the learned
contextual relationships are integrated into semantic labeling in a principled
manner.

###  2.1 Video Object Proposals

We start by generating a set of video object proposals w.r.t. to semantic
categories via object detection and temporal association which encode the
long-range spatio-temporal evolution of various object features. Video object
proposals are commonly tied to higher-level contexts such as object
interactions and behaviours [42, 28, 43, 44, 34, 5, 45].

Specifically, we utilize [46] to extract generic object proposals from each
frame of the input video, and run fast R-CNN [47] on this pool of object
proposals to detect objects w.r.t. the given semantic label. We keep a set of
object hypotheses $\mathbb{D}$ by removing the proposals with detection
confidence lower than a threshold (0.5 in our system). Similar to [5],
temporal association is applied on this cohort of object hypotheses
$\mathbb{D}$ to generate tracks of video object proposals $\mathbb{T}$.
Specifically, we utilize object tracker [48] to track object hypotheses over
time to both ends of the video sequence as follows. We firstly rank all
remaining object hypotheses in $\mathbb{D}$ based on detection confidence;
tracking is performed to both directions initialized by the bounding box of
the highest ranked object hypothesis; object hypothesis in the new frame is
selected and added to $T_{i}$ if it has a sufficient overlap, _i.e_.
Intersection-over-Union (IoU) higher than a threshold (0.5), with the tracker
box; this object hypothesis is consequently removed from $\mathbb{D}$. This
process is iteratively performed until no new trajectory hypothesis containing
three or more object instances can be generated from $\mathbb{D}$. Superpixels
[49] are extracted from each frame as the atomic data units. Let
$\mathcal{R}_{D}$ be the set of superpixels constituting video object
hypotheses, and $\mathcal{R}_{U}$ be the unlabeled superpixels. The parameters
of [49] are set to produce fine superpixels in order to preserve object
boundaries.

###  2.2 Graph Construction

We derive our context model by initializing a k-nearest neighbor similarity
graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ between all $N$ superpixels from
$\mathcal{R}_{D}\cup\mathcal{R}_{U}$. Each node $v_{i}\in\mathcal{V}$ of the
graph is described by the L2-normalized _fc6_ feature $f_{i}$ of VGG-16 Net
[50] in a forward-pass of the corresponding superpixel. The weight
$w_{i,j}\in\mathbf{W}$ of edge $e_{i,j}\in\mathcal{E}$ is computed as the
inner-product between the feature vectors of neighboring nodes, i.e.,
$w_{i,j}=<f_{i},f_{j}>$.

###  2.3 Context Modeling

We start the context modeling process by generating context exemplars. We
consider frames which contain video object proposals as weak annotations,
since the proposals normally capture essential parts of video objects. Let
$\mathcal{F}$ be this set of annotated frames and $\mathcal{\hat{F}}$ be all
the other frames in current video sequence. A context exemplar consists of a
pair of superpixels and the corresponding semantic labels. The intuition
behind this setting is that one superpixel with its semantic label supports
the paired superpixel to be labeled with its corresponding semantic label.
This exemplar is able to encode the global interaction and co-occurrence of
semantic objects beyond local spatial adjacencies. The goal is to impose the
consistency between each pair of superpixels from un-annotated frames and the
extracted context exemplars.

Formally, given a set of semantic labels

$\mathcal{L}=\\{l_{0},l_{1},\dots,l_{C-1}\\}$

comprising all $C$ classes in the annotated frame, we represent the context
exemplars for each class pair $(c_{m},c_{n})$ as

$\mathbf{Q^{m,n}}=\\{(v_{i},v_{j}):L(v_{i})=l_{m},L(v_{j})=l_{n},v_{i},v_{j}\in\mathcal{F}\\}$

where $v_{i},v_{j}\in\mathcal{F}$ stands for two superpixels $v_{i}$ and
$v_{j}$ from the annotated frame set $\mathcal{F}$ and $L(v_{i})$ represents
the semantic label of superpixel $v_{i}$. Hence, all object class pairs as
well as contextual relationships in the annotated frames are represented as

$\mathcal{Q}=\\{\mathbf{Q}^{0,0},\mathbf{Q}^{0,1},\dots,\mathbf{Q}^{C-1,C-1}\\}.$

We transform the above context exemplar to a context link view, where context
exemplar $(v_{i},v_{j})$ can be referred to as a $(l_{m},l_{n})$-type link
between two nodes on the similarity graph. Let $\mathcal{H}$ denote the set of
$N\times N$ matrices, where a matrix $\mathbf{H}^{m,n}\in\mathcal{H}$ is
associated with all $(l_{m},l_{n})$ class pair links. Each entry
$[\mathbf{H}^{m,n}]_{i,j}\in\mathbf{H}^{m,n}$ indicates the confidence of
$(l_{m},l_{n})$-link between two superpixels $v_{i}$ and $v_{j}$. The
confidence indicates the probability of the existence of a link. The
$(l_{m},l_{n})$-links which have been observed within the annotated frames can
be represented by another set of matrices $\mathbf{P}^{m,n}\in\mathcal{H}$
such that

$[\mathbf{P}^{m,n}]_{i,j}=\left\\{\begin{array}[]{lll}1&\mbox{if}&(v_{i},v_{j})\in\mathbf{Q^{m,n}}\\\
0&\mbox{otherwise}&\end{array}\right.$ (1)

All the observed context link can be denoted as

$\mathcal{P}=\\{\mathbf{P}^{0,0},\mathbf{P}^{0,1},\dots,\mathbf{P}^{C-1,C-1}\\}.$

###  2.4 Context Propagation

We formulate the context prediction problem as a link prediction problem which
infers how probable a certain link exists in a graph. Specifically, we predict
$(l_{m},l_{n})$-links among the pairs of nodes from $\mathcal{R}_{U}$ based on
$\mathbf{P}^{m,n}$ consistent to the intrinsic structure of the similarity
graph. To this end, we propagate $(l_{m},l_{n})$-links in $\mathbf{P}^{m,n}$
to estimate the strength of the pairs of nodes from $\mathcal{R}_{U}$. We drop
the $m,n$ suffix for clarity.

Algorithm 1 Context learning algorithm

1:procedure Link prediction

2: $S(v_{i},l_{i},v_{j},l_{j})\leftarrow\varnothing$

3: $\text{Graph}~{}\mathcal{G}\leftarrow\text{all}~{}\textit{superpixels of
video}$

4: $\text{Affinity matrix}~{}\mathbf{W}\leftarrow\textit{k-nearest neighbors}$

5: $d_{i}=\sum_{j=1}^{N}w_{ij}$

6: $\mathbf{D}\leftarrow\mathrm{diag}([d_{1},\dots,d_{N}])$

7:
$\mathbf{L}\leftarrow\mathbf{D}^{-\frac{1}{2}}\mathbf{W}\mathbf{D}^{-\frac{1}{2}}$

8: $\mathcal{Q}\leftarrow\textit{context exemplars}$

9: $\mathcal{P}\leftarrow\textit{context links}\in\mathcal{Q}$

10: for each class pair $(l_{m},l_{n})$ do

11:
$\mathbf{H}_{r}(1)\leftarrow\mathbf{0},~{}\mathbf{H}_{c}(1)\leftarrow\mathbf{0}$

12: $\textit{Convergence}\leftarrow\mathbf{false}$

13: while $\textit{Convergence}~{}\text{is}~{}\mathbf{false}$ do
$\triangleright$ row-wise

14:
$\mathbf{H}_{r}(t+1)\leftarrow\mu\mathbf{L}\mathbf{H}_{r}(t)+(1-\mu)\mathbf{P}^{m,n}$

15: $\textit{Convergence}\leftarrow\mathbf{false}$

16: while $\textit{Convergence}~{}\text{is}~{}\mathbf{false}$ do
$\triangleright$ column-wise

17:
$\mathbf{H}_{c}(t+1)\leftarrow\mu\mathbf{L}\mathbf{H}_{c}(t)+(1-\mu)\mathbf{\hat{H}}_{r}$

18:
$S(v_{i},l_{i}=l_{m},v_{j},l_{j}=l_{n})\leftarrow[\mathbf{\hat{H}}_{c}]_{ij}$

It is impractical to directly solve the link prediction problem for video
segmentation due to the prohibited complexity $O(N^{4})$, and thus we propose
to solve it by decomposing it into two separate label propagation processes.
As described in Algorithm 1, row-wise link predication (step 13-14) is firstly
performed, followed by column-wise link prediction (step 16-17). Specifically,
the $j$-th row $\mathbf{P}^{j,.}$, _i.e_. the context exemplars associated
with $v_{j}$, serves as an initial configuration of a label propagation
problem [51] with respect to node $v_{j}$. Each row is handled separately as a
binary label propagation which converges to $\mathbf{\hat{H}}_{r}$. It is
observed that the label propagation does not apply to the rows of $\mathbf{P}$
corresponding to $\mathcal{R}_{U}$, and thus we only perform row-wise link
propagation in rows corresponding to annotated superpixels, which is much less
than $N$. For the column-wise propagation, the $i$-th converged row
$[\mathbf{\hat{H}}_{r}]_{i}$ is used to initialize the configuration. After
convergence of the column-wise propagation, the probability of
$(l_{m},l_{n})$-link between two nodes of $\mathcal{R}_{U}$ is obtained.

###  2.5 Semantic Labeling

Semantic video object segmentation problem can be formulated as a superpixel
labeling problem. We now describe how to integrate the predicted contextual
relationship into the semantic labeling process. Since we have learned context
link scores $S(v_{i},l_{i},v_{j},l_{j})$, it is straight forward to
incorporate them to the fully connected CRF which is proved to be effective in
encoding model contextual relationships between object classes.

We define a random field $\mathbf{x}$ defined over a set of variables
$\\{\mathrm{x}_{0},\dots,\mathrm{x}_{N-1}\\}$, and the domain of each variable
is a set of class labels $\mathcal{L}=\\{l_{0},l_{1},\dots,l_{C-1}\\}$. The
corresponding Gibbs energy is

$E(\mathbf{x})=\sum_{i}\psi(\mathrm{x}_{i})+\sum_{i,j}\phi(\mathrm{x}_{i},\mathrm{x}_{j}).$
(2)

The pairwise potential $\phi(\mathrm{x}_{i},\mathrm{x}_{j})$ encodes the
contextual relationships between the superpixels learned via link prediction,
which is defined as

$\phi(\mathrm{x}_{i},\mathrm{x}_{j})=\exp(-\frac{S(v_{i},c_{i},v_{j},c_{j})^{2}}{2\beta})$
(3)

where $\beta=<S(v_{i},l_{i},v_{j},l_{j})^{2}>$ is the adaptive weight and
$<\cdot>$ indicates the expectation.

The unary potential $\psi(\mathrm{x}_{i})$ is defined as the negative
logarithm of the likelihood of assigning $v_{i}$ with label $\mathrm{x}_{i}$.
To obtain $\psi(\mathrm{x}_{i})$, we learn a SVM model based on hierarchical
CNN features [48] by sampling from the annotated frames.

We adopt $\alpha$-expansion [52] to optimize (2) and the resulting label
assignment gives the semantic object segmentation of the video sequence.

##  3 Experiments

We evaluate our proposed approach on YouTube-Objects [53] which is the _de
facto_ benchmark for assessing semantic video object segmentation algorithms.
YouTube-Objects contains over 30,000 frames in total with dense annotations
for 20 classes of PASCAL VOC 2012. These videos are very challenging and
completely unconstrained, with objects of similar colour to the background,
fast motion, non-rigid deformations, and fast camera motion. Standard average
IoU is used to measure the segmentation accuracy, $IoU=\frac{S\cap G}{S\cup
G}$, where $S$ is the segmentation result and $G$ stands for the ground-truth
mask. We compare our approach with $8$ state-of-the-art automatic approaches
on this dataset, including two motion driven segmentation [54, 55], three
weakly supervised semantic segmentation approaches [53, 56, 2], and detection
based approaches [3, 5, 57].

Table 1: Intersection-over-union overlap accuracies on YouTube-Objects Dataset

| [54] | [53] | [55] | [56] | [2] | [3] | [5] | [57] | Ours
---|---|---|---|---|---|---|---|---|---
Plane | 0.539 | 0.517 | 0.674 | 0.178 | 0.758 | 0.744 | 0.703 | 0.757 | 0.762
Bird | 0.196 | 0.175 | 0.625 | 0.198 | 0.608 | 0.721 | 0.631 | 0.766 | 0.767
Boat | 0.382 | 0.344 | 0.378 | 0.225 | 0.437 | 0.585 | 0.659 | 0.666 | 0.702
Car | 0.378 | 0.347 | 0.670 | 0.383 | 0.711 | 0.600 | 0.625 | 0.758 | 0.747
Cat | 0.322 | 0.223 | 0.435 | 0.236 | 0.465 | 0.457 | 0.497 | 0.624 | 0.659
Cow | 0.218 | 0.179 | 0.327 | 0.268 | 0.546 | 0.612 | 0.701 | 0.720 | 0.719
Dog | 0.270 | 0.135 | 0.489 | 0.237 | 0.555 | 0.552 | 0.532 | 0.671 | 0.669
Horse | 0.347 | 0.267 | 0.313 | 0.140 | 0.549 | 0.566 | 0.524 | 0.526 | 0.593
Mbike | 0.454 | 0.412 | 0.331 | 0.125 | 0.424 | 0.421 | 0.554 | 0.547 | 0.605
Train | 0.375 | 0.250 | 0.434 | 0.404 | 0.358 | 0.367 | 0.411 | 0.392 | 0.411
Avg. | 0.348 | 0.285 | 0.468 | 0.239 | 0.541 | 0.562 | 0.584 | 0.643 | 0.664

Fig. 1: Qualitative results of our algorithm on YouTube-Objects Dataset.

As summerized in Table 1, our proposed algorithm outperforms the compared
methods, with a considerable margin of $2.1\%$ in average over the best
competing method [57]. In terms of category-wise comparison, our approach
surpasses $8$ out of $10$ categories. This improvement over the state-of-the-
art is owing to the capability of learning and propagating higher-level
spatial-temporal contextual relationships of video object proposals which
capture the key semantic contexts in challenging video data, as opposed to
imposing contextual information in local labeling ([57]) or modeling local
appearance ( [2, 5]). One common limitation of the these methods is that they
are error-prone in separating interacting objects exhibiting similar
appearance or motion, which is intractable unless the inherent contextual
relationship is explored.

Our algorithm outperforms another two methods which also utilize object
detection, _i.e_. [3] and [2], with large margins of $10.2\%$ and $12.3\%$.
[3] uses R-CNN for the initial object detection and incorporates the detection
scores in a graph cut optimization, which, however, does not explore the
semantic contextual information. [2] performs the worst since it only conducts
temporal association of detected object segments without explicitly modeling
either the objects or contexts.

The approaches which train discriminative classifiers given weakly annotated
videos, _i.e_. [55] and [56], perform relatively worse than detection based
approaches. The trained discriminative classifiers can be easily overfitted to
the small amount of annotated data, which makes them difficult to tackle fast-
changing natural scenes. As a contrast, our approach is able to exploit
semantic contextual relationships to compensate the limited capabilities of
object detection on object with unusual viewpoints. Some qualitative results
of the proposed algorithm on YouTube-Objects dataset are shown in Fig. 1.

##  4 Conclusion

We have proposed a novel approach to modeling the semantic contextual
relationships for tackling the challenging video object segmentation problem.
The proposed model comprises an exemplar-based nonparametric view of
contextual cues, which is formulated as link prediction problem solved by
label propagation on a similarity graph of superpixels. The derived contextual
relationships are utilized to estimate the pairwise contexts between all pairs
of unlabeled local superpixels. The experiments demonstrated that modeling the
semantic contextual relationships effectively improved segmentation robustness
and accuracy which significantly advanced the state-of-the-art on challenging
benchmark.

## References

  * [1] Brian Taylor, Alper Ayvaci, Avinash Ravichandran, and Stefano Soatto,  “Semantic video segmentation from occlusion relations within a convex optimization framework,”  in EMMCVPR. Springer, 2013, pp. 195–208.
  * [2] Yu Zhang, Xiaowu Chen, Jia Li, Chen Wang, and Changqun Xia,  “Semantic object segmentation via detection in weakly labeled video,”  in CVPR, 2015, pp. 3641–3649.
  * [3] Benjamin Drayer and Thomas Brox,  “Object detection, tracking, and motion segmentation for object-level video segmentation,”  arXiv preprint arXiv:1608.03066, 2016.
  * [4] Huiling Wang, Tapani Raiko, Lasse Lensu, Tinghuai Wang, and Juha Karhunen,  “Semi-supervised domain adaptation for weakly labeled semantic video object segmentation,”  in ACCV, 2016, pp. 163–179.
  * [5] Tinghuai Wang,  “Submodular video object proposal selection for semantic object segmentation,”  in ICIP, 2017.
  * [6] Guosheng Lin, Chunhua Shen, Anton van den Hengel, and Ian Reid,  “Efficient piecewise training of deep structured models for semantic segmentation,”  in CVPR, 2016, pp. 3194–3203.
  * [7] Spyros Gidaris and Nikos Komodakis,  “Object detection via a multi-region and semantic segmentation-aware cnn model,”  in ICCV, 2015, pp. 1134–1142.
  * [8] Peng Bai, Xinggang Tang, Xiang Wang, and Wenyu Liu,  “Multiple instance detection network with online instance classifier refinement,”  2017\.
  * [9] Tinghuai Wang, Jean-Yves Guillemaut, and John Collomosse,  “Multi-label propagation for coherent video segmentation and artistic stylization,”  in 2010 IEEE International Conference on Image Processing. IEEE, 2010, pp. 3005–3008.
  * [10] Xiaojuan Qi, Renjie Liao, Jiaya Jia, Sanja Fidler, and Raquel Urtasun,  “3d graph neural networks for rgbd semantic segmentation,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 5199–5208.
  * [11] Tinghuai Wang,  “Submodular video object proposal selection for semantic object segmentation,”  in 2017 IEEE International Conference on Image Processing (ICIP). IEEE, 2017, pp. 4522–4526.
  * [12] Wenguan Wang, Xiankai Lu, Jianbing Shen, David J Crandall, and Ling Shao,  “Zero-shot video object segmentation via attentive graph neural networks,”  in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 9236–9245.
  * [13] Tinghuai Wang and Huiling Wang,  “Graph-boosted attentive network for semantic body parsing,”  in Artificial Neural Networks and Machine Learning–ICANN 2019: Image Processing: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17–19, 2019, Proceedings, Part III 28. Springer, 2019, pp. 267–280.
  * [14] Yiding Yang, Zhou Ren, Haoxiang Li, Chunluan Zhou, Xinchao Wang, and Gang Hua,  “Learning dynamics via graph neural networks for human pose estimation and tracking,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 8074–8084.
  * [15] Tinghuai Wang, Huiling Wang, and Lixin Fan,  “Robust interactive image segmentation with weak supervision for mobile touch screen devices,”  in Proceedings of International Conference on Multimedia and Expo. IEEE, 2015, pp. 1–6.
  * [16] Tinghuai Wang and Huiling Wang,  “Method and an apparatus for automatic segmentation of an object,” May 5 2016,  US Patent App. 14/930,392.
  * [17] Yifan Xing, Tong He, Tianjun Xiao, Yongxin Wang, Yuanjun Xiong, Wei Xia, David Wipf, Zheng Zhang, and Stefano Soatto,  “Learning hierarchical graph neural networks for image clustering,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3467–3477.
  * [18] Tinghuai Wang, John Collomosse, David Slatter, Phil Cheatle, and Darryl Greig,  “Video stylization for digital ambient displays of home movies,”  in Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering, 2010, pp. 137–146.
  * [19] Tinghuai Wang, John Collomosse, and Adrian Hilton,  “Wide baseline multi-view video matting using a hybrid markov random field,”  in 2014 22nd International Conference on Pattern Recognition. IEEE, 2014, pp. 136–141.
  * [20] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu,  “Fine-grained video-text retrieval with hierarchical graph reasoning,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10638–10647.
  * [21] Rui Hu, Stuart James, Tinghuai Wang, and John Collomosse,  “Markov random fields for sketch based video retrieval,”  in Proceedings of the 3rd ACM conference on International conference on multimedia retrieval, 2013, pp. 279–286.
  * [22] Tinghuai Wang, John P. Collomosse, David Slatter, Phil Cheatle, and Darryl Greig,  “Video stylization for digital ambient displays of home movies,”  in NPAR, 2010, pp. 137–146.
  * [23] Tinghuai Wang, John Collomosse, Rui Hu, David Slatter, Darryl Greig, and Phil Cheatle,  “Stylized ambient displays of digital media collections,”  Computers & Graphics, vol. 35, no. 1, pp. 54–66, 2011.
  * [24] Tinghuai Wang, John P Collomosse, Andrew Hunter, and Darryl Greig,  “Learnable stroke models for example-based portrait painting.,”  in BMVC, 2013.
  * [25] Huiling Wang and Tinghuai Wang,  “Boosting objectness: Semi-supervised learning for object detection and segmentation in multi-view images,”  in ICASSP, 2016, pp. 1796–1800.
  * [26] Tinghuai Wang,  “Apparatus, a method and a computer program for image processing,” Nov. 15 2016,  US Patent 9,495,755.
  * [27] Gangming Zhao, Weifeng Ge, and Yizhou Yu,  “Graphfpn: Graph feature pyramid network for object detection,”  in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 2763–2772.
  * [28] Tinghuai Wang and Huiling Wang,  “Graph transduction learning of object proposals for video object segmentation,”  in ACCV, 2014, pp. 553–568.
  * [29] Huiling Wang, Tinghuai Wang, Ke Chen, and Joni-Kristian Kämäräinen,  “Cross-granularity graph inference for semantic video object segmentation,”  in IJCAI, 2017, pp. 4544–4550.
  * [30] Tinghuai Wang, Guangming Wang, Kuan Eeik Tan, and Donghui Tan,  “Spectral pyramid graph attention network for hyperspectral image classification,”  arXiv preprint arXiv:2001.07108, 2020.
  * [31] Tinghuai Wang and Lixin Fan,  “Watermark embedding techniques for neural networks and their use,” Jan. 16 2020,  US Patent App. 16/508,434.
  * [32] Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn,  “Generative scene graph networks,”  in International Conference on Learning Representations, 2021.
  * [33] Tinghuai Wang, Huiling Wang, and Lixin Fan,  “A weakly supervised geodesic level set framework for interactive image segmentation,”  Neurocomputing, vol. 168, pp. 55–64, 2015.
  * [34] Huiling Wang and Tinghuai Wang,  “Primary object discovery and segmentation in videos via graph-based transductive inference,”  Computer Vision and Image Understanding, vol. 143, pp. 159–172, 2016.
  * [35] Tinghuai Wang,  “Method, apparatus and computer program product for segmentation of objects in media content,” Apr. 25 2017,  US Patent 9,633,446.
  * [36] Tinghuai Wang,  “Method for analysing media content,” Apr. 26 2018,  US Patent App. 15/785,711.
  * [37] Lingyu Zhu, Tinghuai Wang, Emre Aksu, and Joni-Kristian Kamarainen,  “Portrait instance segmentation for mobile devices,”  in ICME, 2019, pp. 1630–1635.
  * [38] Lingyu Zhu, Tinghuai Wang, Emre Aksu, and Joni-Kristian Kamarainen,  “Cross-granularity attention network for semantic segmentation,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019, pp. 0–0.
  * [39] Tinghuai Wang,  “Semantic segmentation based on a hierarchy of neural networks,” Dec. 22 2020,  US Patent 10,872,275.
  * [40] Xiankai Lu, Wenguan Wang, Martin Danelljan, Tianfei Zhou, Jianbing Shen, and Luc Van Gool,  “Video object segmentation with episodic graph memory networks,”  in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer, 2020, pp. 661–679.
  * [41] Tao Wang, Ning Xu, Kean Chen, and Weiyao Lin,  “End-to-end video instance segmentation via spatial-temporal graph neural networks,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10797–10806.
  * [42] Tinghuai Wang and John P. Collomosse,  “Probabilistic motion diffusion of labeling priors for coherent video segmentation,”  IEEE Transactions on Multimedia, vol. 14, no. 2, pp. 389–400, 2012\.
  * [43] Tinghuai Wang, Bo Han, and John P. Collomosse,  “Touchcut: Fast image and video segmentation using single-touch interaction,”  Computer Vision and Image Understanding, vol. 120, pp. 14–30, 2014\.
  * [44] Buyu Liu and Xuming He,  “Multiclass semantic video segmentation with object-level active inference,”  in CVPR, 2015, pp. 4286–4294.
  * [45] Peng Tang, Chunyu Wang, Xinggang Wang, Wenyu Liu, Wenjun Zeng, and Jingdong Wang,  “Object detection in videos by short and long range object linking,”  arXiv preprint arXiv:1801.09823, 2018.
  * [46] Ian Endres and Derek Hoiem,  “Category independent object proposals,”  in ECCV, 2010, pp. 575–588.
  * [47] Ross Girshick,  “Fast r-cnn,”  in ICCV, 2015, pp. 1440–1448.
  * [48] Chao Ma, Jia-Bin Huang, Xiaokang Yang, and Ming-Hsuan Yang,  “Hierarchical convolutional features for visual tracking,”  in ICCV, 2015, pp. 3074–3082.
  * [49] Pedro F Felzenszwalb and Daniel P Huttenlocher,  “Efficient graph-based image segmentation,”  International journal of computer vision, vol. 59, no. 2, pp. 167–181, 2004.
  * [50] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [51] Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Sch,  “Learning with local and global consistency,”  in NIPS, 2004, pp. 321–328.
  * [52] Yuri Boykov, Olga Veksler, and Ramin Zabih,  “Fast approximate energy minimization via graph cuts,”  IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 11, pp. 1222–1239, 2001.
  * [53] Alessandro Prest, Christian Leistner, Javier Civera, Cordelia Schmid, and Vittorio Ferrari,  “Learning object class detectors from weakly annotated video,”  in CVPR, 2012, pp. 3282–3289.
  * [54] Thomas Brox and Jitendra Malik,  “Object segmentation by long term analysis of point trajectories,”  in ECCV, 2010, pp. 282–295.
  * [55] Anestis Papazoglou and Vittorio Ferrari,  “Fast object segmentation in unconstrained video,”  in ICCV, 2013, pp. 1777–1784.
  * [56] Kevin D. Tang, Rahul Sukthankar, Jay Yagnik, and Fei-Fei Li,  “Discriminative segment annotation in weakly labeled video,”  in CVPR, 2013, pp. 2483–2490.
  * [57] Huiling Wang, Tinghuai Wang, Ke Chen, and Joni-Kristian Kämäräinen,  “Cross-granularity graph inference for semantic video object segmentation,”  in IJCAI, 2017, pp. 4544–4550.