# S2Former-OR: Single-Stage Bimodal Transformer for Scene Graph Generation in
OR

Jialun Pei  Member, IEEE  Diandian Guo  Jingyang Zhang  Manxi Lin  Yueming Jin
Member, IEEE
Pheng-Ann Heng  Senior Member, IEEE J. Pei, D. Guo, J. Zhang, and P.-A. Heng
are with the Department of Computer Science and Engineering, P.-A. Heng is
also with the Institute of Medical Intelligence and XR, The Chinese University
of Hong Kong, HKSAR, China. (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>jyzhang, pheng@cse.cuhk.edu.hk)M. Lin is with the
Department of Applied Mathematics and Computer Science, Technical University
of Denmark, Kongens Lyngby, Denmark. (email: manli@dtu.dk)Y. Jin is with the
Department of Electrical and Computer Engineering, National University of
Singapore, Singapore.(e-mail: ymjin@nus.edu.sg)Jialun Pei and Diandian Guo
contributed equally.Corresponding author: Yueming Jin.

###### Abstract

Scene graph generation (SGG) of surgical procedures is crucial in enhancing
holistically cognitive intelligence in the operating room (OR). However,
previous works have primarily relied on the multi-stage learning that
generates semantic scene graphs dependent on intermediate processes with pose
estimation and object detection, which may compromise model efficiency and
efficacy, also impose extra annotation burden. In this study, we introduce a
novel single-stage bimodal transformer framework for SGG in the OR, termed _S
2Former-OR_, aimed to complementally leverage multi-view 2D scenes and 3D
point clouds for SGG in an end-to-end manner. Concretely, our model embraces a
View-Sync Transfusion scheme to encourage multi-view visual information
interaction. Concurrently, a Geometry-Visual Cohesion operation is designed to
integrate the synergic 2D semantic features into 3D point cloud features.
Moreover, based on the augmented feature, we propose a novel relation-
sensitive transformer decoder that embeds dynamic entity-pair queries and
relational trait priors, which enables the direct prediction of entity-pair
relations for graph generation without intermediate steps. Extensive
experiments have validated the superior SGG performance and lower
computational cost of _S 2Former-OR_ on 4D-OR benchmark, compared with current
OR-SGG methods, _e.g._ , 3% Precision increase and 24.2M reduction in model
parameters. We further compared our method with generic single-stage SGG
methods with broader metrics for a comprehensive evaluation, with consistently
better performance achieved. The code will be made available.

Scene graph generation, 3D surgical scene understanding, Transformer, single-
stage, bimodal.

##  1 Introduction

As medicine and information technology advance, the operating room (OR) has
evolved into a highly complicated and technologically rich environment. This
setting is characterized by high variability and dynamic, unpredictable
scenarios [1]. Surgical procedures in the modern OR are increasingly complex,
necessitating intricate coordination among medical personnel, patients, and a
multitude of devices [2]. An automatic and holistic comprehension of surgical
procedures is critically essential for monitoring and optimizing surgical
processes, scheduling surgeons, and enhancing coordination among surgical
teams [3]. To this end, holistically understanding OR can augment OR
efficiency, improve patient safety, and enhance the quality of patient
treatment [4]. Scene graph generation (SGG), a powerful semantic
representation technique derived from the computer vision field, has shown
great potential in capturing complex interactive relationships between
entities in the 3D dynamic OR environment [4, 5]. Effective SGG facilitates a
comprehensive understanding of the entire OR scene, thus highly desirable for
promoting cognitive assistance for the next generation of operating
intelligence.

Figure 1: Diagrams of multimodal architectures for scene graph generation in
operating rooms. (a) Existing multi-stage model [6]; (b) Our proposed single-
stage model in an end-to-end manner.

To advance SGG methods for OR modeling, Özsoy et al. [6] introduced the first
OR-SGG dataset, termed 4D-OR, which contains multi-view video frames and
corresponding 3D point clouds in the knee surgery theatre. Moreover, they
proposed a multi-stage semantic SGG pipeline that provides 3D human pose
estimation and object detection, followed by blending different forms of
prediction labels and constructing a scene graph through a 3D SGG framework
[7]. Subsequently, LABRAD-OR [5] proposed to enhance the multi-stage framework
by incorporating temporal information into scene graphs. The constructed
memory scene graphs can capture fine-grained relationships between entities in
dynamic OR, achieving state-of-the-art results. However, existing methods rely
on a multi-stage model training (shown in Fig. 1 (a)), typically facing the
following challenges which may hinder its applicability in real-world
deployment: i) In terms of efficiency, multi-stage approaches require step-
wise processing, which imposes additional annotation burdens and computational
complexity during model training. Meanwhile, the inherent structure of a
multi-stage framework elongates inference time, potentially compromising the
ability to detect abnormalities in OR promptly. ii) In terms of precision, the
intermediate representations generated at each stage may not fully retain the
contextual information desired for scene modeling, consequently failing to
capture the long-range dependencies within the scene and risking losing
representational information. To overcome these limitations, we aim at the
_single-stage_ paradigm that properly utilizes both 2D images and 3D point
cloud modalities to generate OR scene graphs.

Numerous generic SGG methods have demonstrated significant success in general
computer vision tasks, _e.g._ , graph generation of natural scenes. These
methods primarily consist of two main streams: 2D and 3D SGG models. However,
both of them exhibit shortcomings when being directly adapted to the surgical
domain. The OR scene is more complicated, composed of entities that vary
significantly in size and shape—for instance, surgeons compared to anesthesia
equipment and surgical instruments. Moreover, these entities are frequently
occluded due to the common presence of multiple medical staff members who
typically surround the patient throughout the surgical procedure. The existing
2D SGG models [8, 9, 10, 11, 12, 13] though can precept natural scenes
accurately, for complex surgical procedures, these SGG models lack geometric
information reinforcement to provide a comprehensive perception of the
associations between people and medical devices.

On the other hand, typical 3D SGG approaches [7, 14, 15] on point cloud data
assume the availability of 3D instance labels as known inputs. This assumption
significantly increases annotation efforts and is often impractical in a
clinical setting. Moreover, surgical actions performed by the chief surgeon
are very subtle, making it difficult for 3D SGG methods to precisely capture
specific operations without sufficient semantic information. In this regard,
how to efficiently leverage geometric information and productively dig out 2D
semantic cues is crucial for accurate scene graph generation in the OR.
Concentrating on this issue, we strive to design a concise multimodal SGG
paradigm for OR scenarios.

In this paper, we present a novel single-stage multi-view bimodal transformer-
based SGG model, termed _S 2Former-OR_, for efficient scene graph generation
in OR recordings. As illustrated in Fig. 1 (b), our single-stage end-to-end
SGG paradigm enables the efficient generation of scene graphs, providing a
more concise and effective solution for scene understanding in surgery. In
contrast to multi-stage OR-SGG methods [6, 5], our model requires no
integration of intermediate predicted instances and has no reliance on the
supervision of pose estimation labels. Concretely, we propose a relation-
sensitive transformer in the relation decoder of _S 2Former-OR_. This
structure first couples the output embeddings produced by the transformer
decoder into a set of subject-object pairs and then incorporates relational
trait priors to generate dynamic relation queries. By interacting with multi-
view memories through cross-attention, our model predicts relationships within
subject-object pairs and achieves the final scene graph. To better converge
and exploit features from different modalities, a geometry-visual cohesion
(GVC) module is introduced to assemble 2D multi-view semantic cues and 3D
geometric features to generate appearance-structure representations, instead
of the general projection operation in most existing works. In the multi-view
unit, we also present a view-sync transfusion (VST) scheme that complements
the local fine-level information of multiple views to the main view features
to enhance the holistic visual semantic representation and mitigate
occlusions. Extensive experimental findings demonstrate the effectiveness of
‘single stage’.

Our main contributions are summarized as follows:

  * •

To our best knowledge, _S 2Former-OR_ is the first single-stage bimodal
framework for SGG in the operating rooms. The proposed OR-SGG model can
directly generate accurate scene graphs with end-to-end training.

  * •

We develop a geometry-visual cohesion (GVC) operation, which integrates 2D
synergic semantic features and 3D point cloud geometric features to unified
visual-structure representations. Besides, a view-sync transfusion (VST)
scheme is designed to consolidate multi-view visual appearance information.

  * •

A relation-sensitive transformer is introduced to perceive relations between
subject-object pairs and directly generate scene graphs. Wherein, we propose
dynamic relation queries using trait priors and then interacting with multi-
view features to exploit potential associations.

  * •

Extensive validations demonstrate that our _S 2Former-OR_ achieves superior
performance on the 4D-OR benchmark, whether compared to multi-stage OR-SGG
methods or to generic single-stage SGG models.

##  2 Related Work

###  2.1 Semantic Scene Graph in OR

The main literature in surgical data science focuses on analyzing the surgical
scene, such as surgical workflow and tool recognition [16], instrument
segmentation [17], and scene understanding [18]. Scene understanding in
dynamic operating room (OR) environments is a new challenging task, which
builds entity interactive systems by semantically modeling the holistic space
[6, 5]. Özsoy et al. [6] first revealed the challenge of SSG generation for
the OR scene. Specifically, they built a new scene graph dataset (4D-OR) for
knee surgery scenarios, which consists of multi-view RGB frames and point
cloud data accompanied by human pose labels, 3D object bounding boxes, and
scene graph labels. Accordingly, they also proposed a multi-stage SGG
baseline, which first predicts the human pose and object proposals by
VoxelPose [19] and Group-Free [20] respectively, and then combines the
prediction labels and feeds them into the 3DSSG model [7] for scene graph
prediction. Building upon the 4D-OR baseline, LABRAD-OR [5] is a novel
lightweight SGG model that introduces the concept of memory scene graphs. This
model fuses temporal cues with visual features to improve the SGG accuracy.
However, both approaches are based on multi-stage architectures, while real
clinical scenarios require more on the real-time performance that these models
lack, instead of their multi-stage outputs. To this end, we propose a single-
stage bimodal SGG framework. As depicted in Fig. 1, our model takes multi-view
inputs and point cloud data to directly generate scene graphs without
intermediate processing, enhancing the practicality of OR-SGG models in real-
world surgery.

Figure 2: Overview of the proposed single-stage multi-view bimodal _S 2Former-
OR_ for scene graph generation from operating rooms. We first extract
appearance and geometric features separately based on 2D multi-view images and
3D point cloud inputs. In the multi-view unit, a View-Sync Transfusion (VST)
is suggested for synthesizing multi-view semantic features. Then, we introduce
Geometry-Visual Cohesion (GVC) to fuse 2D synergic features and 3D point cloud
features to obtain unified features. In the entity unit, we utilize the
unified features and entity queries to predict entity proposals; in the
relation unit, we generate dynamic relation queries by assembling latent
entity pairs with relational trait priors, which are fed into our relation-
sensitive transformer to generate scene graphs in the operating theatre.

###  2.2 2D Scene Graph Generation

The main purpose of Scene Graph Generation (SGG) is to construct
representations of visual relationships between objects, which has been
extensively studied in the field of computer vision in 2D images [4]. In the
deep learning era, mainstream SGG approaches [10, 8, 9, 11] primarily adopt a
two-stage framework, firstly locating objects using an object detection
network, and then employing various mechanisms to purify contextual structural
information and refine location-specific features, _e.g._ , message passing
[8], attention module [11, 21], and visual translation [22], to predict
relationships by classifiers and generate scene graphs. Recently, inspired by
fully convolutional network (FCN) in object detection models [23], one-stage
SGG methods employ FCN-based [24, 25] or CNN-Transformer architectures [12,
13] to efficiently predict visual relationships directly from global features
using sparse proposal sets and queries, making this paradigm concise and easy
to converge. To further improve performance, subsequent one-stage models, such
as SGTR [12] and RelTR [13], utilize the Transformer architecture and the
attention mechanism to capture global perception and predict triplets by
interacting with subject-object queries. However, 2D images lack critical
geometric information for SGG in ORs. For better capturing location-related
relationships, our model incorporates 3D point cloud inputs that complement 2D
semantic information from the images to enhance the perception of structural
correlations.

###  2.3 3D Scene Graph Generation in Point Cloud

In contrast to 2D SGG at the image level, feeding point clouds to 3D SGG
models provides a more robust representation of the geometric properties of
objects [15], leading to a higher sensitivity in predicting structure-aware
relationships. Wald et al. [7] developed a large-scale 3D semantic scene graph
(SSG) dataset, termed 3DSSG. They also proposed an end-to-end SGG pipeline for
constructing 3D scene graphs via graph convolution networks (GCNs). Later,
Zhang et al. [14] introduced an approach for directly generating scene graphs
from 3D point clouds, which encodes the point clouds into more discriminative
latent spaces as a priori knowledge. Nevertheless, it is challenging to learn
semantic information from the point cloud, since the point cloud is defined on
geometric structures instead of more semantic colors or textures. For this
reason, VL-SAT [15] incorporates 2D instance labels for multimodal training,
which assists the 3D-only SGG model in capturing richer semantic information.
It is worth noting that the 3DSSG benchmark generally assumes the instance
labels as known inputs during model training. In contrast, for the 4D-OR
benchmark, instance labeling in the operating theatre is difficult and
laborious to acquire. This drives us to design a bimodal SGG model that
complements 3D modalities by 2D semantics to generate a comprehensive scene
understanding without the need for known instance labels.

##  3 Methodology

The overall architecture of the proposed single-stage bimodal _S 2Former-OR_
is described in Fig. 2. For the input of four view images, we employ a CNN-
Transformer encoder to extract multi-view semantic features. Subsequently, we
introduce a view-sync transfusion (VST) scheme that complements the multi-view
information into the main view to attain comprehensive and enriched synergic
features. Bilaterally, for 3D point cloud input, we adopt PointNet++ [26] as
the point encoder to capture abundant geometric features. Given 2D and 3D
modal representations, we design a geometry-visual cohesion (GVC) operation to
implicitly interact synergic semantic information with point cloud features to
achieve unified appearance-structure features. After that, we leverage the
unified feature to yield entity (_i.e._ , subject and object) proposals and
predict relationships. In the operating room, the entities include surgeons,
patients, operating tables, instruments, etc. In the entity unit, we
initialize a set of entity queries and produce entity proposals with the
instance decoder in line with DETR [27]. In order to predict relationships
without relying on predictions from intermediate stages, we propose a
relation-sensitive transformer decoder to perceive interactions between entity
pairs and generate OR scene graphs. We will detail the structure of VST, GVC,
and relation-sensitive transformers embedded in _S 2Former-OR_ in the
following subsections.

###  3.1 Multi-View and Bimodal Interaction

_S 2Former-OR_ comprises two encoder units to process multi-view 2D images and
3D point cloud data of the whole scene, respectively. To utilize the multi-
view representation to comprehensively emphasize surgical operations and
enhance structure perception through holistic point cloud samples, we
introduce a view-sync transfusion (VST) scheme to synergize multi-view
features and a geometry-visual cohesion (GVC) operation to complement the
geometric information.

####  3.1.1 Multi-View Unit with VST

In the multi-view image encoder, we utilize four different camera viewpoints
{$I_{i}$}${}_{i=1}^{4}$ as the 2D inputs. The multi-view images are initially
passed through ResNet-50 [28] to extract semantic features. Following the
encoder design of DETR [27], we flatten the last layer features $R_{i}^{5}$
from ResNet-50 for each viewpoint. We leverage a transformer encoder to
enhance the global perception of different views. The transformer encoder
consists of six layers, containing a multi-head self-attention module and a
feed-forward network (FFN) in each layer. After the shape-restoring operation,
we can produce high-level semantic features for all four viewpoints. While
_View_ #1 provides a broader receptive field, several critical surgical
operations (_e.g._ , hammering, drilling, cutting) require specific viewpoints
to focus on localized situations. Hence, it is essential to synergize features
from more viewpoints for a comprehensive understanding of the whole OR.

With this in mind, we propose a VST scheme in the multi-view unit to integrate
multi-view features extracted from the transformer encoder, generating the
synergic feature $F_{s}$. Among features from the four viewpoints, _View_ #1
embraces a more expansive scene, thus we consider it as _Main View_ while the
other views (_i.e._ , _View_ #2, _View_ #3, and _View_ #4) serve as auxiliary
viewpoints to complement the main view. Inspired by the cross-attention
operation in transformers [29, 30], we introduce a multi-view cross-
interaction module to integrate shared context information between the main
and auxiliary views. Specifically, we treat the flattened main-view feature
$F_{m}\in\mathbb{R}^{wh\times 256}$ as the Query. In counterpart, the features
of _View_ #2, _View_ #3, and _View_ #4 ($F_{a}^{j},j=2,3,4$) are concatenated
and flattened to form an auxiliary viewpoint vector
$F_{\alpha}\in\mathbb{R}^{3wh\times 256}$, serving as the Key and Value
$F_{\alpha}=[F_{a}^{2},...,F_{a}^{j}]\in\mathbb{R}^{3wh\times 256}$. Through
the multi-view cross-interaction operation, we implement the implicit
interaction between global features from the main view and other auxiliary
views. This operation can be formulated as

$\mathcal{A}(F_{m},F_{\alpha})=Softmax(\frac{W_{q}F_{m}(W_{k}F_{\alpha})^{T}}{\sqrt{d_{k}}})W_{v}F_{\alpha},$
(1)

where $W_{q}$, $W_{k}$, and $W_{v}$ represent linear mapping matrices and
$d_{k}$ is the dimension of $F_{m}$ after linear transformation. Then, the
aggregated features are performed with the main view features for a residual
connection and layer normalization to enhance holistic information, followed
by an FFN and layer normalization to generate the synergic feature
$F_{s}\in\mathbb{R}^{wh\times 256}$.

####  3.1.2 Point Cloud Unit with GVC

In parallel, to enrich the geometric information available for improving the
structural relationship prediction, we add 3D point cloud inputs to our point
encoder unit. For an input point cloud $P\in\mathbb{R}^{N\times 3}$ with $N$
points, we employ PointNet++ [26] as the 3D backbone to sample and group the
input point cloud to a feature space of 256 dimension and 1024 points,
resulting in the point cloud feature $F_{p}\in\mathbb{R}^{1024\times 256}$.

After obtaining 2D synergic features and 3D point cloud features, we can
generate complementary information from both modalities for sufficient
integration of appearance and geometric cues. Previous works [6, 5] utilize 2D
and 3D samples for pose estimation and object detection, respectively, and
then combine the predicted instance labels of humans and objects to generate
scene graphs via GCN [7]. To enable our single-stage framework for training
efficiency while leveraging 3D representation, we inject point cloud features
into the synergic feature to facilitate supervision by only utilizing image-
level ground truth. In this regard, we design the GVC operation to implicitly
heterogeneously cohesion the fused multi-view synergic features with the 3D
point cloud features rather than the direct projection manipulation.
Specifically, the synergic 2D feature $F_{s}$ and the point cloud feature
$F_{p}$ are aligned in the self-attention module at different feature levels,
respectively. Subsequently, the cross-attention operation is employed to
interact and blend features from different modalities, where the Key and Value
are both point cloud feature vectors and the Query is the synergic 2D feature
vector. Finally, a residual connection is applied to consolidate semantic
information. The entire process of GVC can be described as follows:

$F_{u}=F_{s}+\mathcal{CA}(\mathcal{SA}(F_{s}),\mathcal{SA}(F_{p})),$ (2)

where $\mathcal{SA}$ and $\mathcal{CA}$ stand for self-attention and cross-
attention operations, respectively. The unified feature $F_{u}$ comprises
sufficient 2D semantic cues and 3D structural information for predicting
entities and generating scene graphs.

###  3.2 Entity Pair Generation

####  3.2.1 Entity Predictions

Before predicting relationship triplets [entitysub, relation, entityobj] which
contain the _relation_ between _subject entity_ and _object entity_ in the
operating theatre, we first generate entity boxes in the entity unit, allowing
for the subsequent pairing of entities to predict the relationships in the
surgical scene. Following DETR [27], we utilize a set of randomly initialized
queries as input to the transformer decoder. In the transformer decoder, we
interact the unified feature $F_{u}$ with queries to produce the output
embedding via cross-attention operations. Then an FFN is used to get entity
proposals with class labels. During the training phase, we supervise the
prediction of entity boxes by 2D box labels. It is worth noting that box
supervision is trained together with the final scene graph labels without
additional pre-training.

####  3.2.2 Dynamic Relation Queries

The previous multi-stage SGG-OR models [6, 5] first predict different types of
instance labels (_i.e._ , human pose estimation labels and object detection
labels), and then merge them through intermediate steps before feeding them
into the subsequent SGG network. In contrast, our framework proposes to
establish entity connections after generating entity proposals and then
predicts potential relationships between entity pairs. Inspired by the concept
of building edges between nodes in graph convolutional networks (GCNs), in
relation unit, we construct all potential subject-object pairs. For the
subject and object in each pair, we extract semantic features, spatial
features, and supplemental point cloud features within the entity regions to
generate relational trait priors and then yield dynamic relation queries (see
Fig. 3). Specifically, we first filter out $N$ entities with class scores
greater than 0.5 and pair them to generate $N(N-1)$ entity pairs. Here the
number of $N$ is not fixed and is dynamically changed according to the
perception of entity proposals in the entity unit. To obtain corresponding
semantic features, we concatenate the entity-aware embeddings associated with
entities. Spatial features are calculated based on the positional attributes
between entities, including differences in centroid coordinates, distances
between the coordinates of center points, and bounding box areas. Given a pair
of entities with center point coordinates $(E_{x}^{i},E_{y}^{i})$ and
$(E_{x}^{j},E_{y}^{j})$, the spatial feature $S_{ij}$ can be computed as

$\bar{x}=E_{x}^{i}-E_{x}^{j},\quad\bar{y}=E_{y}^{i}-E_{y}^{j},$ (3)
$S_{ij}=[\bar{x},\bar{y},\sqrt{\bar{x}^{2}+\bar{y}^{2}}),Ai,Aj],$ (4)

where $Ai$ and $Aj$ are areas of two entity boxes, respectively. Furthermore,
we also embed the supplemental point cloud features to enhance the structural
representation. We then concatenate the three kinds of entity features and use
a multi-layer perception (MLP) to obtain relational trait priors:

$V_{ij}=MLP([R_{ij},S_{ij},P_{ij}])\in\mathbb{R}^{1\times 256},$ (5)
$Q_{r}=[V_{1},...,V_{N(N-1)}]\in\mathbb{R}^{N(N-1)\times 256},$ (6)

where $R_{ij}$ denotes the semantic feature, $P_{ij}$ is the point cloud
feature, and $V_{ij}$ indicates the relational trait prior for per entity
pair. Then, we concatenate the relational trait priors of all entity pairs to
generate dynamic relational queries $Q_{r}$.

Figure 3: Illustration of our relation-sensitive transformer decoder.

###  3.3 Relation-Sensitive Transformer

Unlike previous works [31, 6] that predict relationships via GCN, we input
dynamic relation queries into the relation-sensitive transformer block to
predict predicates between subject-object pairs. As illustrated in Fig. 3, the
dynamic relational query $Q_{r}$ is first fed into the self-attention module
to enhance the relational representation of each triplet, obtaining
intermediate features $Q_{r}^{{}^{\prime}}$. Considering that relation queries
are insensitive to location information during self-attention operation, we do
not input position encoding [29] into the transformer block. After that, we
utilize cross-attention operation to interact $Q_{r}^{{}^{\prime}}$ with 2D
semantic features. Here, we use $Q_{r}^{{}^{\prime}}$ as the Query and the
unified feature $F_{u}$ is regarded as the encoder memory feature for the Key
and Value. Since (_View_ #4) concentration on the operation relationship
between the surgeon and the patient, we also incorporate (_View_ #4) features
into the memory features to optimize predictions of some action relationships
(_e.g._ , touching, suturing, and cementing). As shown in Fig. 2, we
concatenate the _View_ #4 features extracted from the last layer of ResNet
with the unified feature: $\mathcal{M}$=$[F_{u},R_{4}]\in\mathbb{R}^{2wh\times
256}$. Combining CNN features essentially serves to supplement encoder memory
features with lower-level features. The memory feature $\mathcal{M}$ is used
as the Key and Value. Sec. 4.3 discusses the effectiveness of this operation
in detail with the ablation study. Additionally, to improve the efficiency of
cross-attention, inspired by masked attention from Mask2Former [32], we retain
only the union of foreground regions of entities for the attention operation.
The relation cross-attention operation can be expressed as

$\mathcal{A}(Q_{r}^{{}^{\prime}},\mathcal{M})=Softmax(\frac{W_{q}Q_{r}^{{}^{\prime}}(W_{k}\mathcal{M})^{T}+\mathcal{Z}}{\sqrt{d_{k}}})W_{v}\mathcal{M},$
(7) $\mathcal{Z}=\begin{cases}0,&\text{if}\quad(x,y)\in U\\\
-\infty,&\text{otherwise}\end{cases},$ (8)

where $\mathcal{Z}$ indicates the attention mask and $U$ denotes the union of
foreground regions. $(x,y)$ represents the pixel coordinates. Following a
residual connection, layer normalization, and an FFN, we can obtain the
predicted triples and combine them to generate a scene graph.

###  3.4 Objective Function

We train our model in an end-to-end manner. The total objective function of _S
2Former-OR_ consists of the entity box loss $L_{ent}$ and the relation
classification loss $L_{rel}$ in the relationship-sensitive decoder:

$\mathcal{L}_{total}=\lambda\mathcal{L}_{ent}+\mathcal{L}_{rel},$ (9)

where $\lambda$ is the weighting factor. In line with DETR, $L_{ent}$ contains
L1 loss and GIoU loss [33] for box regression and Focal loss [34] ($L_{foc}$)
for entity classification. Here, $\lambda$ is set to 1. In addition, we add
the auxiliary loss [35] after each decoder layer. The relational
classification loss is mainly implemented by Focal loss:

$\mathcal{L}_{rel}=\frac{1}{\sum_{i=1}^{C}g_{i}}\sum_{i=1}^{C}g_{i}L_{foc}(\hat{g_{i}},g_{i}),$
(10)

where $C$ is the number of relation classes. $\hat{{g}_{i}}$ is the predicted
probability of the $i$-th relation, and $g_{i}\in\\{0,1\\}$ indicates whether
the label includes the relation category or not.

##  4 Experiments

###  4.1 Dataset and Evaluation Metrics

####  4.1.1 Dataset

We used a large-scale public benchmark dataset 4D-OR [6] for method
validation. As a new yet essential task of OR-SGG, to our best knowledge, only
4D-OR dataset [6] is currently publicly available yet well-known to be widely
used for validation. This dataset consists of ten knee replacement surgery
procedures with a total of 6,734 scenes. Each scene provides 2D multi-view
images from different viewpoints and a 3D point cloud description. The ten
surgical workflows are divided into three sets, with six takes (4,024 scenes)
used for training, two takes (1,332 scenes) used for validation, and two takes
(1,378 scenes) used for testing111The test set is not publicly available and
requires using the online addresses https://bit.ly/4D-OR_evaluator to evaluate
SGG results. The 4D-OR dataset contains rich annotations, including scene
entities with 12 categories in the operating room and relationship labels with
14 categories. In addition, they also provide 3D entity boxes and 6D human
pose annotations for auxiliary supervision.

Since the entity box labels are utilized in 2D format for our single-stage
training, in our experiments, we project the 3D box labels annotated in 4D-OR
into 2D boxes via camera parameters. Similar to the 4D-OR method [6], to
compensate for the recognition of medical instruments, we locate the hands of
the head surgeon by the left and right wrist coordinates from pose
annotations. Then, we mark boxes with 100$\times$100 size to produce the
instrument annotations. The 2D supervision labels generated from ready-made
annotations will be available at https://github.com/PJLallen/S2Former-OR.

####  4.1.2 Evaluation Metrics

Consistent with the state-of-the-art methods [6, 5], we used precision,
recall, and Macro F1 metrics to evaluate predicted scene graph relationships.
We also employed the same evaluation metrics as [6, 5] in the downstream task
of clinical role prediction. Moreover, in comparison with the generic SGG
models on the 4D-OR benchmark, we adopted several widely used metrics for
generic SGG model evaluation, including recall metric under the IoU threshold
of 0.5 (R$@$50), weighted mean average precision of relationship detection
(wmAPrel), and phrase detection (wmAPphr) from Open Images Challenge [36] for
a more comprehensive evaluation.

###  4.2 Implementation Details

We train the proposed model for 80 epochs on 2 RTX3090 GPUs with an AdamW
optimizer. The batch size per GPU is set to 2, and the initial learning rate
is 5e-5 with a weight decay of 0.0001. We adopt ResNet-50 [28] and the DETR
encoder [27] pretrained on MS-COCO as the multi-view image encoder. The point
cloud encoder and other components are randomly initialized. During training,
the short edges of the input multi-view images are randomly resized between
480 and 800 with aspect ratio unchanged. Additionally, we apply color jitter
and random horizontal flip for data augmentation. During validation and
inference, the short edges are fixed to 480 with no data augmentation. In the
transformer and relation-sensitive decoders, we use six multi-head attention
layers with 256 hidden dimensions. The number of entity queries is set to 20
and the IoU threshold in NMS is 0.7.

Table 1: Detailed comparisons with the state-of-the-art OR-SGG methods on 4D-OR test set. Stage | Methods | Params | Metrics | Assist | Cement | Clean | CloseTo | Cut | Drill | Hammer | Hold | LyingOn | Operate | Prepare | Saw | Suture | Touch | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Multi-Stage | 4D-OR [6] | 84.8M | Precision | 0.42 | 0.78 | 0.53 | 0.97 | 0.49 | 0.87 | 0.71 | 0.55 | 1.00 | 0.55 | 0.62 | 0.69 | 0.60 | 0.41 | 0.68
Recall | 0.93 | 0.78 | 0.63 | 0.89 | 0.49 | 1.00 | 0.89 | 0.95 | 0.99 | 0.99 | 0.91 | 0.91 | 1.00 | 0.69 | 0.87
Macro F1 | 0.58 | 0.78 | 0.57 | 0.93 | 0.49 | 0.93 | 0.79 | 0.70 | 0.99 | 0.71 | 0.74 | 0.79 | 0.75 | 0.51 | 0.75
Multi-Stage | LABRAD-OR [5] | 85.8M | Precision | 0.60 | 0.96 | 0.86 | 0.96 | 0.91 | 1.00 | 0.93 | 0.71 | 1.00 | 0.85 | 0.77 | 0.78 | 1.00 | 0.71 | 0.87
Recall | 0.86 | 0.93 | 0.72 | 0.94 | 0.68 | 0.94 | 0.95 | 0.95 | 1.00 | 0.99 | 0.91 | 0.93 | 1.00 | 0.72 | 0.90
Macro F1 | 0.71 | 0.94 | 0.78 | 0.95 | 0.78 | 0.97 | 0.94 | 0.81 | 1.00 | 0.91 | 0.84 | 0.85 | 1.00 | 0.71 | 0.88
Single-Stage | _S 2Former-OR_ | 60.6M | Precision | 0.65 | 1.00 | 0.91 | 0.97 | 0.82 | 0.98 | 0.94 | 0.79 | 1.00 | 0.90 | 0.83 | 0.93 | 0.94 | 0.81 | 0.90
Recall | 0.66 | 0.94 | 0.86 | 0.93 | 0.85 | 0.92 | 0.94 | 0.83 | 1.00 | 0.91 | 0.89 | 0.99 | 0.97 | 0.49 | 0.88
Macro F1 | 0.66 | 0.97 | 0.89 | 0.95 | 0.84 | 0.95 | 0.94 | 0.81 | 1.00 | 0.91 | 0.86 | 0.96 | 0.95 | 0.61 | 0.89

###  4.3 Comparison with State-of-the-Arts

To evaluate the performance of our _S 2Former-OR_ for the OR-SGG task, we
carried out detailed comparisons with existing SGG approaches tailored to OR
scenes and generic single-stage 2D SGG models on the 4D-OR benchmark. For a
fair comparison, we uniformly use the official code to train and evaluate each
model on 4D-OR validation and test sets.

Figure 4: Qualitative results of the 4D-OR model [6], LABRAD-OR [5], and our
_S 2Former-OR_ on the 4D-OR validation set. The blue rectangles represent the
human attribute and the red ellipses denote the object attribute.
Correct/Erroneous predicted relationships are shown in green/red.

####  4.3.1 Comparison with OR-SGG Methods

We conducted a comprehensive comparison between the proposed _S 2Former-OR_
and existing OR-SGG models, _i.e._ , 4D-OR [6] and LABRAD-OR [5]. Table 1
exhibits quantitative results of our model and competitors in terms of the
precision, recall, and macro F1 metrics. We can observe that _S 2Former-OR_
demonstrates promising performance, particularly in average precision and
macro F1 score, achieving 0.90 and 0.89, respectively. Benefiting from the
thorough fusion of semantic features and structural information, our model
excelled in predicting critical surgeon-patient actions, _e.g._ , achieving
Macro F1 scores of 0.97, 0.91, and 0.96 in cementing, operating, and sawing
relations, respectively. More importantly, _S 2Former-OR_ acts as an end-to-
end model with fewer parameters, enabling the direct generation of scene
graphs in practical applications by feeding surgical data without intermediate
prediction processes (_e.g._ , no need for pose estimation labels). As a
single-stage architecture, the inference time of our model is also faster than
the 4D-OR model by 0.3s per frame, significantly increasing the potential of
model deployment in real clinical practice.

To further intuitively evaluate the effectiveness of generated scene graphs,
we visualize the prediction results of some keyframes in Fig. 4. Notably, we
omitted the “CloseTo” relationship to explicitly display the scene graph that
is most relevant to the surgical procedure. We can see that our model
consistently predicts the interactions of entities without missing key
messages in complex operating room environments. Compared to state-of-the-art
SGG-OR models, our model exhibits more accurate detection of entities,
avoiding the influence of erroneous entities in predicting relationships.
While still producing redundant predictions in some cases, our model can
predict the overall structure of the scene graph more precisely and robustly
compared with others.

####  4.3.2 Comparison with Single-Stage SGG Methods

To further validate the robustness of our method, we also compare with the
cutting-edge single-stage SGG methods in a generic scenario of natural scene
generation (_i.e._ , SGTR [12] and RelTR [13]). We reimplemented them on the
4D-OR dataset. Both of them are novel single-modal end-to-end architectures
based on transformers. As shown in Table 2, our model not only achieves better
results in vanilla metrics but also outperforms other competitors in R$@$50,
wmAPrel, and wmAPphr metrics. This can be attributed to the interaction of
multi-view features via the proposed VST scheme and the extraction of
geometric information from point cloud modalities by our GVC operation.
Further, the sufficient utilization of relational trait priors in generating
dynamic relation queries and the efficient relation-sensitive transformer
structure have also brought performance gains to our model.

Table 2: Quantitative comparisons with generic single-stage SGG methods on the 4D-OR validation set. Methods | Backbones |  R$@$50 |  wmAPrel |  wmAPphr | Prec | Rec | F1
---|---|---|---|---|---|---|---
SGTR [12] | Res101 | 0.57 | 0.51 | 0.69 | 0.69 | 0.70 | 0.69
RelTR [13] | Res50 | 0.54 | 0.48 | 0.62 | 0.61 | 0.67 | 0.63
_S 2Former-OR_ | Res50 | 0.71 | 0.63 | 0.83 | 0.90 | 0.84 | 0.87

###  4.4 Ablation Study

####  4.4.1 Effectiveness of Each Component

To validate the effectiveness of each component in our model, we conduct
detailed ablation experiments to optimize model performance. The main
components include View-Sync Transfusion (VST), Geometry-Visual Cohesion
(GVC), and Relation-sensitive Transformer (RST). Note that when ablating the
VST module, we replace the input of multi-view images with only the main-view
image. When removing the GVC module, we directly project the extracted point
cloud features to synergic features through the intrinsic and extrinsic
matrices provided in the dataset. For our RST part, we adopt a GCN structure
from the previous OR-SGG model [6] as a replacement to perform relation
predictions. The results in Table 3 indicate that each proposed component
imposes a positive impact on _S 2Former-OR_. Compared to the baseline, our
model gains 0.19, 0.24, and 0.22 improvement on the precision, recall, and
macro F1 metrics after adding three essential parts (refer to the first and
sixth rows). The third row of Table 3 illustrates that the GVC operation
improves the generated accuracy by sufficiently integrating point cloud
features into 2D semantic features. Moreover, the fifth row highlights the
indispensability of RST, where the dynamic relation queries and relation-aware
cross-attention operations make our model more conducive to inferring
relations. Compared to other variants, _S 2Former-OR_ has no substantial
increase in parameters after embedding all components, while achieving better
performance.

Table 3: Ablation studies for each component in our _S 2Former-OR_ on the 4D-OR validation set, including View-Sync Transfusion (VST), Geometry-Visual Cohesion (GVC), and Relation-sensitive Transformer (RST). ID | VST | GVC | RST | Prec | Rec | F1 | Params
---|---|---|---|---|---|---|---
1 |  |  |  | 0.71 | 0.60 | 0.65 | 54.1M
2 |  |  | ✓ | 0.84 | 0.67 | 0.71 | 54.6M
3 | ✓ |  | ✓ | 0.84 | 0.68 | 0.72 | 57.2M
4 |  | ✓ | ✓ | 0.89 | 0.83 | 0.86 | 58.0M
5 | ✓ | ✓ |  | 0.80 | 0.71 | 0.75 | 60.1M
6 | ✓ | ✓ | ✓ | 0.90 | 0.84 | 0.87 | 60.6M
Table 4: Comparison with different numbers of relation queries on the 4D-OR validation set. Number | Prec | Rec | F1
---|---|---|---
20 | 0.85 | 0.75 | 0.78
50 | 0.86 | 0.83 | 0.84
100 | 0.87 | 0.83 | 0.85
300 | 0.88 | 0.84 | 0.85
Dynamic | 0.90 | 0.84 | 0.87

####  4.4.2 Number of Relation Queries

Relation queries are an essential component in our relation-sensitive
transformer. To examine the impact of the number of relation queries on
relation prediction, we compare the performance of our dynamic relation
queries (_i.e._ , $N(N-1)$) with other fixed number queries (_i.e._ , 20, 50,
100, and 300). Unlike dynamic queries, to produce a fixed number of queries,
we directly pair the entity proposals output in the entity unit without using
category scores for suppression. Then, we apply an MLP operation to all entity
pairs to obtain corresponding confidence values and use the top-K selection to
filter out a fixed number of pairs for relation-aware query generation. As
shown in Table 4, with a fixed number of relation queries, the performance of
_S 2Former-OR_ constantly increases with the number of queries until the
number reaches 100. Compared to the fixed-amount setting, our dynamic relation
queries provide greater adaptability to variable scene interactions, making
our relation-sensitive transformer more accurate in graph prediction, _e.g._ ,
0.87 vs. 0.85 in macro F1.

####  4.4.3 Effect of Relational Trait Priors

In the relation-sensitive decoder, we inject the latent entity pairs with
various subject-object trait priors to accurately express relation queries,
including semantic, spatial, and point cloud features. To validate the
effectiveness of each feature representation, we compare the contributions of
different combinations of prior trait embeddings on the 4D-OR validation set.
As illustrated in Table 5, in addition to the indispensable semantic features,
the spatial feature representation of entity pairs remarkably improves the
precision value by 3%. It indicates that spatial features can connect entity
pairs with the closest correlations. Additionally, point cloud features also
contribute to enhancing the structural representation of entity pairs. By
aggregating all relational trait priors, _S 2Former-OR_ reaches the best
performance.

Table 5: Combinations of prior trait embeddings in relation queries. Relational Traits | Prec | Rec | F1
---|---|---|---
Semantic | 0.86 | 0.82 | 0.84
Semantic + Spatial | 0.89 | 0.83 | 0.86
Semantic + Spatial + Point Cloud | 0.90 | 0.84 | 0.87

####  4.4.4 Memory Feature in Relation-Sensitive Transformer

Table 6: Ablations for combinations of cross-attention memories in relation-sensitive transformer. Memories | Prec | Rec | F1
---|---|---|---
$F_{u}$ | 0.80 | 0.61 | 0.67
$F_{u}$ \+ $R_{4}$ | 0.90 | 0.84 | 0.87
$F_{u}$ \+ $R_{2}$ \+ $R_{4}$ | 0.89 | 0.85 | 0.86
$F_{u}$ \+ $R_{2}$ \+ $R_{3}$ \+ $R_{4}$ | 0.87 | 0.87 | 0.87

Relation queries involve interacting with memory features in the proposed
relation-sensitive transformer to perceive relationships between entities in
ORs. As discussed in Sec. 3.3, we also added low-level CNN features from other
viewpoints to capture crucial actions. To investigate the effect of different
combinations of cross-attention memories on relationship prediction, we
attempt to embed features from additional viewpoints to explore the upper
bound of the performance of our relation-sensitive transformer. As shown in
Table 6, the combination of the unified feature $F_{u}$ and the _View_ #4
feature $R_{4}$ from ResNet gains better performance. Memories combined with
more view features have not further contributed to the relationship
predictions. The main reason is that _View_ #4 concentrates on the view of
surgeon-patient interaction, and thus better assisting the main view in
complementing the emphasis on the surgical-operative relationship. Besides,
the CNN features also provide lower-level cues to unified features.

Fig. 5 visualizes the attention maps from cross-attention weights in the
relation-sensitive transformer. The top row shows that memory features from
the main view can interact comprehensively with entity regions, while the
bottom row illustrates that features from _View_ #4 enhance attention toward
the surgeon’s actions (_e.g._ , hammering and holding). Specifically, from the
third column in Fig. 5, we can see that when the unified features from the
main view fail to accurately focus on the instrument region, the _View_ #4
attention map still can capture the position of “holding” for complementation.
Whereas, in cases where the unified feature effectively highlights the
relationship between entity pairs, assistance from other views is unnecessary.
Overall, combining memory features of $F_{u}$ and $R_{4}$ yields superior
performance for relationship predictions.

Figure 5: Visualizations of attention maps in our relation-sensitive
transformer on the 4D-OR validation set. The top row displays the main view
weight map with the unified feature $F_{u}$; the bottom row displays the
View#4 weight map with the ResNet feature $R_{4}$.

###  4.5 Clinical Role Prediction

Assigning role labels to humans in surgical scenes facilitates further
improving surgical process monitoring and optimization. Since the 4D-OR
benchmark provides role labels for each entity associated with the human
attribute (_i.e._ , Patient, Head Surgeon, Assistant Surgeon, Circulating
Nurse, and Anaesthetist), we perform clinical role prediction based on the
generated scene graphs. For a fair comparison, we also adopt Graphormer [37]
to infer the clinical role probabilities of human entities as in the 4D-OR
method [6]. Since our model is a single-stage framework without intermediate
results of human pose estimation, we select the predicted entities belonging
to the human attribute as human nodes. Fig. 6 compares the performance of our
model with 4D-OR on the downstream task of clinical role prediction. It
indicates that _S 2Former-OR_ achieves strong performance in all clinical
roles, especially in the ‘Anaesthetist’ and ‘Circulating Nurse’ categories,
improving the macro F1 score by 14% and 5%. Since the anaesthetist is usually
located in the corner of operating rooms and appears exclusively at the
beginning stage of surgery, there is still room for further improvement.
Overall, our model carries over the advantages of scene graph generation to
clinical role prediction.

##  5 Discussion

Scene graph generation (SGG) in the operating room (OR) is essential for
monitoring the surgical process, optimizing team collaboration, and improving
workflow efficiency. Compared to reviewing surveillance records, generating
scene graphs can highlight specific events of the surgical process and save
the time cost of identifying abnormal events. However, it is challenging to
generate scene graphs directly from multimodal data streams. Existing OR-SGG
approaches [6, 5] first detect human poses and objects, which are then used as
inputs to predict scene graphs via GCNs. Although this multi-stage
architecture reaches good performance, it is complicated and difficult to
deploy in real clinical practices. Driven by the above motivations, this paper
aims to infer SGG in ORs using 2D multi-view images and 3D point cloud data
without intermediate steps of processing.

Figure 6: Performance comparison of clinical role prediction between our _S
2Former-OR_ and the 4D-OR model on the 4D-OR dataset.

We propose a novel single-stage multi-view bimodal transformer (_S 2Former-
OR_) model for SGG in ORs. In practice, our model is constructed primarily
based on a 2D SGG baseline augmented by point cloud samples. In complex
surgical procedures, leveraging point clouds to enhance geometric perception
is crucial for prediction robustness. Besides, the data streams from multiple
cameras are also essential for monitoring specific actions of the head surgeon
and overall scene recognition. Therefore, our single-stage model includes a
View-Sync Transfusion (VST) module for synthesizing multi-view semantic
features and a Geometry-Visual Cohesion (GVC) operation for injecting
structural cues extracted from point cloud features into synergic features. To
generate scene graphs in an end-to-end manner, we embed a relation-sensitive
transformer in _S 2Former-OR_, which utilizes unified features and entity-
aware embeddings to predict subject-object relations.

As the only benchmark for OR SGG, the 4D-OR dataset [6] contains a variety of
interaction relations (_e.g._ , Assisting, Drilling, and Touching) and spatial
structural relations (_e.g._ , Close-to and Lying-on). Unlike general 3D SGG
datasets [7] that focus primarily on spatial relations, 4D-OR requires more
precise prediction of human-human or human-object interactions [38, 39, 40].
In this regard, our method is dedicated to the fine-grained extraction and
fusion of multi-view semantic features, especially with our VST scheme and
relational trait priors. As shown in Table 1, _S 2Former-OR_ achieves superior
performance in predicting relationships initiated by surgeons, especially some
subtle actions, such as Cementing, Cleaning, Cutting, and Sawing. Such
capability to robustly predict subtle relationships benefits our model to be
applied not only to knee surgery but also to be generalized to other types of
surgery. We believe our method can inspire more investigations on single-stage
scene graph generation in operating rooms, further boosting the efficiency of
surgical process analysis and promoting the development of surgical
intelligence.

Although we contribute a single-stage multimodal OR-SGG framework, there are
still several key points that deserve further exploration in this study.
First, we have not yet fully utilized the temporal information from multi-view
video frames. In future research, we will incorporate efficient temporal
interaction operations into the single-stage pipeline to further improve
temporal consistency. Second, although the efficiency of our method has
largely outperformed previous methods, _i.e._ , 0.3s faster per scene, the
proposed model has not yet reached real-time inference. In future efforts, we
shall further improve the training and inference efficiency with the aid of
pre-trained foundation models and a more lightweight design for cost-effective
application in actual surgical scenes. Furthermore, we consider integrating
the clinical role labels directly into the SGG annotations, which will
automatically recognize the attributes of each human node during scene graph
generation, strengthening the fine-grained modeling for OR scenes. We will
optimize the above-mentioned issues and explore innovative techniques to
further optimize the OR-SGG performance in future work.

##  6 Conclusion

We present a novel single-stage multi-view bimodal transformer (_S 2Former-
OR_) method for scene graph generation in operating rooms. Specifically, we
design a View-Sync Transfusion (VST) scheme to synthesize scene information
from multiple viewpoints and a Geometry-Visual Cohesion (GVC) operation to
implicitly merge 2D semantic features with 3D point cloud geometric cues.
Furthermore, a relation-sensitive transformer is proposed to directly predict
scene graphs via dynamic relation queries and blend multimodal features. The
promising performance on the 4D-OR dataset demonstrates the effectiveness of
our approach. More importantly, compared to existing multi-stage OR-SGG
models, our single-stage end-to-end framework can generate scene graphs
directly from surgical process content, greatly facilitating real-life
applications in clinical surgical scenes.

## References

  * [1] F. Lalys and P. Jannin, “Surgical process modelling: a review,” _Int. J. Comput. Assist. Radiol. Surg._ , vol. 9, no. 3, pp. 495–511, 2014.
  * [2] L. Maier-Hein, S. S. Vedula, S. Speidel, N. Navab, R. Kikinis, A. Park, M. Eisenmann, H. Feussner, G. Forestier, S. Giannarou _et al._ , “Surgical data science for next-generation interventions,” _Nature Biomedical Engineering_ , vol. 1, no. 9, pp. 691–696, 2017.
  * [3] L. R. Kennedy-Metz, P. Mascagni, A. Torralba, R. D. Dias, P. Perona, J. A. Shah, N. Padoy, and M. A. Zenati, “Computer vision in the operating room: Opportunities and caveats,” _IEEE TMRB_ , vol. 3, no. 1, pp. 2–10, 2020\.
  * [4] X. Chang, P. Ren, P. Xu, Z. Li, X. Chen, and A. Hauptmann, “A comprehensive survey of scene graphs: Generation and application,” _IEEE TPAMI_ , vol. 45, no. 1, pp. 1–26, 2021.
  * [5] E. Özsoy, T. Czempiel, F. Holm, C. Pellegrini, and N. Navab, “Labrad-or: Lightweight memory scene graphs for accurate bimodal reasoning in dynamic operating rooms,” in _MICCAI_ , 2023, pp. 302–311.
  * [6] E. Özsoy, E. P. Örnek, U. Eck, T. Czempiel, F. Tombari, and N. Navab, “4d-or: Semantic scene graphs for or domain modeling,” in _MICCAI_ , 2022, pp. 475–485.
  * [7] J. Wald, H. Dhamo, N. Navab, and F. Tombari, “Learning 3d semantic scene graphs from 3d indoor reconstructions,” in _IEEE CVPR_ , 2020, pp. 3961–3970.
  * [8] R. Li, S. Zhang, B. Wan, and X. He, “Bipartite graph network with adaptive message passing for unbiased scene graph generation,” in _IEEE CVPR_ , 2021, pp. 11 109–11 119.
  * [9] X. Lin, C. Ding, J. Zeng, and D. Tao, “Gps-net: Graph property sensing network for scene graph generation,” in _IEEE CVPR_ , 2020, pp. 3746–3753.
  * [10] K. Tang, Y. Niu, J. Huang, J. Shi, and H. Zhang, “Unbiased scene graph generation from biased training,” in _IEEE CVPR_ , 2020, pp. 3716–3725.
  * [11] N. Dhingra, F. Ritter, and A. Kunz, “Bgt-net: Bidirectional gru transformer network for scene graph generation,” in _IEEE CVPR_ , 2021, pp. 2150–2159.
  * [12] R. Li, S. Zhang, and X. He, “Sgtr: End-to-end scene graph generation with transformer,” in _IEEE CVPR_ , 2022, pp. 19 486–19 496.
  * [13] Y. Cong, M. Y. Yang, and B. Rosenhahn, “Reltr: Relation transformer for scene graph generation,” _IEEE TPAMI_ , 2023.
  * [14] S. Zhang, A. Hao, H. Qin _et al._ , “Knowledge-inspired 3d scene graph prediction in point cloud,” _NeurIPS_ , vol. 34, pp. 18 620–18 632, 2021\.
  * [15] Z. Wang, B. Cheng, L. Zhao, D. Xu, Y. Tang, and L. Sheng, “Vl-sat: Visual-linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud,” in _IEEE CVPR_ , 2023, pp. 21 560–21 569.
  * [16] T. Czempiel, M. Paschali, M. Keicher, W. Simson, H. Feussner, S. T. Kim, and N. Navab, “Tecno: Surgical phase recognition with multi-stage temporal convolutional networks,” in _MICCAI_ , 2020.
  * [17] E. Colleoni and D. Stoyanov, “Robotic instrument segmentation with image-to-image translation,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 935–942, 2021.
  * [18] Y. Jin, Y. Yu, C. Chen, Z. Zhao, P.-A. Heng, and D. Stoyanov, “Exploring intra-and inter-video relation for surgical semantic scene segmentation,” _IEEE TMI_ , vol. 41, no. 11, pp. 2991–3002, 2022.
  * [19] H. Tu, C. Wang, and W. Zeng, “Voxelpose: Towards multi-camera 3d human pose estimation in wild environment,” in _ECCV_ , 2020, pp. 197–212.
  * [20] Z. Liu, Z. Zhang, Y. Cao, H. Hu, and X. Tong, “Group-free 3d object detection via transformers,” in _IEEE ICCV_ , 2021, pp. 2949–2958.
  * [21] Y. Lu, H. Rai, J. Chang, B. Knyazev, G. Yu, S. Shekhar, G. W. Taylor, and M. Volkovs, “Context-aware scene graph generation with seq2seq transformers,” in _IEEE ICCV_ , 2021, pp. 15 931–15 941.
  * [22] H. Zhang, Z. Kyaw, S.-F. Chang, and T.-S. Chua, “Visual translation embedding network for visual relation detection,” in _IEEE CVPR_ , 2017, pp. 5532–5540.
  * [23] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _IEEE CVPR_ , 2015, pp. 3431–3440.
  * [24] H. Liu, N. Yan, M. Mortazavi, and B. Bhanu, “Fully convolutional scene graph generation,” in _IEEE CVPR_ , 2021, pp. 11 546–11 556.
  * [25] E. Özsoy, F. Holm, T. Czempiel, N. Navab, and B. Busam, “Location-free scene graph generation,” _arXiv preprint arXiv:2303.10944_ , 2023.
  * [26] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” _NeurIPS_ , vol. 30, 2017\.
  * [27] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _ECCV_ , 2020, pp. 213–229.
  * [28] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE CVPR_ , 2016, pp. 770–778.
  * [29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _NeurIPS_ , vol. 30, 2017.
  * [30] S. Chen, T. Yu, and P. Li, “Mvt: Multi-view vision transformer for 3d object recognition,” in _BMVC_ , 2021, p. 349.
  * [31] M. Qi, W. Li, Z. Yang, Y. Wang, and J. Luo, “Attentive relational networks for mapping images to scene graphs,” in _IEEE CVPR_ , 2019, pp. 3957–3966.
  * [32] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, “Masked-attention mask transformer for universal image segmentation,” in _IEEE CVPR_ , 2022, pp. 1290–1299.
  * [33] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” in _IEEE CVPR_ , 2019, pp. 658–666.
  * [34] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _IEEE ICCV_ , 2017, pp. 2980–2988.
  * [35] R. Al-Rfou, D. Choe, N. Constant, M. Guo, and L. Jones, “Character-level language modeling with deeper self-attention,” in _AAAI_ , 2019, pp. 3159–3166.
  * [36] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov _et al._ , “The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale,” _IJCV_ , vol. 128, no. 7, pp. 1956–1981, 2020.
  * [37] C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu, “Do transformers really perform badly for graph representation?” _NeurIPS_ , vol. 34, pp. 28 877–28 888, 2021.
  * [38] B. Kim, J. Lee, J. Kang, E.-S. Kim, and H. J. Kim, “Hotr: End-to-end human-object interaction detection with transformers,” in _IEEE CVPR_ , 2021, pp. 74–83.
  * [39] Y. Liao, A. Zhang, M. Lu, Y. Wang, X. Li, and S. Liu, “Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection,” in _IEEE CVPR_ , 2022, pp. 20 123–20 132.
  * [40] Y. Zhang, Y. Pan, T. Yao, R. Huang, T. Mei, and C.-W. Chen, “Exploring structure-aware transformer over interaction proposals for human-object interaction detection,” in _IEEE CVPR_ , 2022, pp. 19 548–19 557.