# Advancing Surgical VQA with Scene Graph Knowledge

Kun Yuan Manasi Kattel Joël L. Lavanchy Nassir Navab Vinkle Srivastav Nicolas
Padoy<EMAIL_ADDRESS>ICube, University of Strasbourg, CNRS, Strasbourg,
France  IHU Strasbourg, Strasbourg, France  CAMP, Technische Universität
München, Munich, Germany

###### Abstract

Purpose: The modern operating room is becoming increasingly complex, requiring
innovative intra-operative support systems. While the focus of surgical data
science has largely been on video analysis, integrating surgical computer
vision with natural language capabilities is emerging as a necessity. Our work
aims to advance Visual Question Answering (VQA) in the surgical context with
scene graph knowledge, addressing two main challenges in the current surgical
VQA systems: removing question-condition bias in the surgical VQA dataset and
incorporating scene-aware reasoning in the surgical VQA model design.

Methods: First, we propose a Surgical Scene Graph-based dataset, SSG-QA,
generated by employing segmentation and detection models on publicly available
datasets. We build surgical scene graphs using spatial and action information
of instruments and anatomies. These graphs are fed into a question engine,
generating diverse QA pairs. Our SSG-QA dataset provides a more complex,
diverse, geometrically grounded, unbiased, and surgical action-oriented
dataset compared to existing surgical VQA datasets. We then propose SSG-QA-
Net, a novel surgical VQA model incorporating a lightweight Scene-embedded
Interaction Module (SIM), which integrates geometric scene knowledge in the
VQA model design by employing cross-attention between the textual and the
scene features.

Results: Our comprehensive analysis of the SSG-QA dataset shows that SSG-QA-
Net outperforms existing methods across different question types and
complexities. We highlight that the primary limitation in the current surgical
VQA systems is the lack of scene knowledge to answer complex queries.

Conclusion: We present a novel surgical VQA dataset and model and show that
results can be significantly improved by incorporating geometric scene
features in the VQA model design. The source code and the dataset will be made
publicly available at: https://github.com/CAMMA-public/SSG-QA

Keywords: Visual question answering, Multi-modality learning, Surgical Data
Science.

##  1 Introduction

Surgical data science is a rapidly growing field that aims to streamline
clinical workflows and enable the development of real-time intra-operative
decision-support systems [16, 18]. Recent advancements in surgical video
analysis, such as surgical workflow phase recognition, fine-grained surgical
action detection, and surgical semantic scene segmentation, show evidence of
the progress [17, 3, 21]. However, the scope of these methods is confined as
it mainly focus on visual-only data to perform classification or recognition
tasks, thereby offering limited user interaction. The next generation of
surgical data science applications also demands approaches operating at the
crucial intersection of vision and language to offer intuitive user
interaction during intra-operative surgical procedures. Surgical Visual
Question Answering (VQA) is emerging as a notable solution in that direction,
which aims to provide precise answers to user queries in a natural language by
analyzing a given surgical image [1, 10, 20, 19].

Fig. 1: SSG-QA dataset contains up to $50$ complex visual reasoning questions,
compared to $2$ classification-based questions in the Cholec-VQA [20].

Developing an effective surgical VQA system is inherently challenging for a
typical surgical scene, which contains multiple anatomical structures and
instruments connected through diverse spatial and action relationships. While
a few works have explored VQA tasks in the surgical context [20, 19], they are
typically limited to datasets and models that ignore detailed scene knowledge.
From the dataset perspective, one key challenge is the lack of a dataset with
potentially a vast set of question-answer pairs covering various aspects of
the surgical scene. The current surgical VQA datasets are small and only
consider simple scene information, e.g., object/action occurrence, as shown in
Fig. 1. Moreover, these datasets contain question-answer pairs with
significant question-condition bias, where answers can be derived from just
the questions without performing any visual processing. This hinders the
utility of these datasets to serve as appropriate surgical VQA benchmarks.

From the model perspective, the current surgical VQA architectures operate on
the global visual representation of the surgical image, ignoring the detailed
understanding of surgical scene knowledge. This can be detrimental, especially
when object-level visual reasoning is essential to answer fine-grained
questions. Our key contributions are therefore twofold: the introduction of a
new surgical scene-aware VQA dataset called SSG-QA and a novel surgical VQA
model called SSG-QA-Net.

The SSG-QA dataset uses a semantic scene graph [14] as a suitable
representation to generate diverse question-answer pairs. A semantic scene-
graph representation provides scene knowledge by detecting objects and their
attributes and connecting relationships and interactions between objects in
the scene. To develop the surgical semantic scene graph, we use publicly
available datasets for semantic segmentation and tool detection [11, 8], and
apply these models to estimate object spatial relationships. We then estimate
the surgical action relationships, i.e., $<instrument,verb,target>$, among the
objects using the public CholecT45 dataset [17].

We then develop a surgery-specific question engine that ingests the surgical
scene graph and manually designed question templates to produce a variety of
question-answer pairs. Including detailed surgical scene understanding along
with question templates helps us to generate question-answer pairs covering
various aspects of the surgical scene, for example, fine-grained action
recognition - “What is the action being performed on peritoneum?”, semantic
scene reasoning - “What anatomy is at the top-mid of the frame?”, and surgical
object attribute reasoning - “What is the name of the anatomy that is being
retracted?”. Furthermore, we perform the balancing and sampling strategies
based on the surgical-specific knowledge and class distribution to remove the
questions that contain question-condition bias, e.g., “How many livers are in
the frame?” which counts the number of certain anatomical structures. The
overall pipeline is illustrated in Fig. 2.

Given a large-scale SSG-QA dataset containing fine-grained surgical question-
answer pairs, we propose a multi-modality surgical VQA model called SSG-QA-
Net. Existing surgical VQA models use a highly parameterized multi-modality
transformer encoder to fuse the textual embeddings from a question and the
patches from a global visual representation of a surgical image [20, 19].
However, these patches do not contain object-wise information about the
surgical scene, hence missing the geometric scene understanding. Our key idea
is to exploit object-wise local features and fuse geometric scene information
in the VQA model design. To enable this, we train a fast and lightweight
object detector, YOLOv7 [22], on the bounding-box labels of the SSG-QA
dataset. The trained object detector allows us to extract object-wise local
representations of the surgical scene objects using RoIAlign pooling [6].
Furthermore, we integrate the geometric spatial coordinates and class labels
of detected bounding boxes into the VQA model by introducing a lightweight
multi-modal transformer encoder named the Scene-embedded Interaction Module
(SIM). The SIM module uses a scene graph of detected bounding boxes where each
node contains the class label and bounding-box coordinate information. The
scene graph is refined by cross-attention between the scene graph and the
textual inputs, highlighting specific graph nodes correlated to the complex
question query. These refined text-aware scene embeddings are then combined
with the object-wise local representations of the surgical scene and the
textual embeddings through a transformer encoder layer to generate an accurate
response. Experimental results show that our method outperforms prior works
while achieving a low parameter count. We summarize our main contributions as
follows:

  * 1.

We present a new surgical scene graph-based VQA dataset, SSG-QA, providing
complex, diverse, geometrically grounded, and surgical action-oriented
question answers.

  * 2.

We present a surgical VQA model, SSG-QA-Net, utilizing a novel scene-aware
feature extraction strategy to obtain state-of-the-art performance.

##  2 Methodology

###  2.1 SSG-QA dataset

In this section, we explain the SSG-QA dataset generation process consisting
of creating surgical scene graphs, designing a question engine with diverse
templates, and employing a sampling strategy to mitigate data imbalance and
question-condition bias.

####  2.1.1 Scene graph generation

We build our SSG-QA dataset using the publicly available CholecT45 [17],
m2cai16-tool-locations [11] and CholecSeg8k [8] datasets. Specifically, we
train a tool detection model [12] on m2cai16-tool-locations [11] and a
semantic segmentation model [4] on CholecSeg8k [8] to extract bounding boxes
of surgical objects, including surgical instruments and anatomies. Then, we
build the surgical semantic scene graph using the detections. A surgical scene
graph can be formulated as a set of nodes and edges, where the nodes represent
surgical objects that contain a set of attributes, i.e., color, location, and
type, and edges represent the spatial and action relations among the objects.
The spatial relations are calculated by comparing the centroid of objects, and
the action relations are provided by the triplet annotations from CholecT45
[17]. Then, we leverage the generated scene graphs as input to a question
engine, as described below, to generate diverse question-answer pairs. Note
that to create a clean test set of question-answer pairs, we manually correct
the bounding boxes and class labels of scene graphs in the test videos.

Fig. 2: Pipeline of SSG-QA construction. The dataset is constructed from the
well-designed question engine, which takes the scene graph as input and
changes the parameters of question templates to generate diverse question-
answer pairs.

####  2.1.2 Question engine

The question engine, which is responsible for generating diverse questions
with various categories, requires two inputs, i.e., scene graph and question
templates. We use the CLEVR engine [13, 10] and extend it to the surgical
context. Specifically, the question engine can change question templates’
parameters conditioned on the surgical scene graph to express diverse
questions. For example, the question “what is the tool to the left of yellow
anatomy?” can be formed by the template “what is the tool $<$R$>$ $<$C$>$
$<$L$>$ $<$T$>$?”, by replacing the parameters $<$R$>$, $<$C$>$, $<$L$>$ and
$<$T$>$ into “to the left of”, “yellow”, “null” and “anatomy”. The questions
are parameterized by five parameters, namely $<$C$>$ (color), $<$L$>$
(location), $<$T$>$ (type), $<$N$>$ (name), and $<$R$>$ (relationship). In
total, there are $40$ question templates containing different types of
questions, such as querying object (e.g., “what is the name of instruments to
the left of the gallbladder?”), querying attribute ( e.g., “there is an object
that is both to the left of the yellow thing and below the brown anatomy; what
is its location?”), querying relation (e.g., “what is the action being
performed?”), confirming existence (e.g., “is there a bipolar in the top-mid
location?”), and counting (e.g., “how many instruments are in the bottom-
right?”).

Generated questions also fall into three categories depending on their
complexity: zero-hop, one-hop, and single-and. Each requires different visual
reasoning steps to resolve. Specifically, solving these three types of
questions involves the understanding of relations between zero, one, or two
surgical objects, respectively. Examples from each category are provided in
the supplementary material. The question engine allows us to freely control
the complexity, length, and number of questions per image.

####  2.1.3 Sampling and balancing

Here, we introduce the applied strategies to reduce the effect of class
imbalance and question-condition bias during SSG-QA dataset construction.
Surgical VQA is a multi-choice task, which mainly includes questions about the
surgical objects. Therefore, an imbalance in the occurrence of surgical
objects could lead to an imbalance in their class distribution. To address
that, we sample the frame amounts based on the surgical phase and tool
presence labels from the Cholec80 dataset, instead of sampling evenly like
Cholec80-VQA [20]. Also, to address the question-condition bias that the
information is leaked out from poorly formulated questions, we remove the
question templates that may contain the question-condition bias. We also
eliminate poorly formulated questions, such as “What is the location of the
$<$N$>$?” with $<$N$>$=gallbladder when there is no gallbladder in the scene.
These processing strategies prevent question-condition bias and avoid
generating degenerate question-answer pairs. The overall pipeline is shown in
Fig. 2.

###  2.2 SSG-QA-Net

####  2.2.1 Pipeline

Fig. 3: Pipeline of the SSG-QA-Net. It requires three types of inputs,
textual, visual, and scene knowledge. The textual and scene embeddings are fed
into the SIM and generate refined scene embeddings. The visual embeddings are
generated from the RoIAlign. Finally, we concat them to feed into the self-
attention transformer to get the final answer. Here, G, H, A, and L represent
class labels; $x_{1}$, $y_{1}$, $x_{2}$, and $y_{2}$ represent bounding box
coordinates (G: gallbladder, H: hook; A: abdominal wall cavity; L: Liver).

Here, we explain the pipeline of SSG-QA-Net. Given the textual form of a
question, we first extract textual embeddings of questions using a pre-trained
tokenizer [20], denoted as $T=\\{t_{1},...t_{K}\\}$. From the surgical scene
image, we extract a feature map using the ResNet18 [7] visual backbone. Then
we use a trained object detector, YOLOv7 [22], to detect the surgical objects
and extract $N$ object-wise visual embeddings using RoIAlign pooling, denoted
as $V=\\{v_{1},...v_{N}\\}$. The object detector is trained on the bounding
boxes of surgical objects from the SSG-QA training dataset.

We build the scene embeddings using the detected surgical objects’
information. Specifically, we initialize the graph nodes as a concatenation of
objects’ class labels and spatial coordinates, as shown in Fig. 3. These low-
dimensional embeddings are projected using a linear layer, called Scene
Encoder, to match the dimensionality of textual embeddings. These scene
embeddings $S=\\{s_{1},...s_{N}\\}$ are then passed through our proposed
Scene-embedded interaction (SIM) module, explained below, to obtain text-aware
scene embeddings ($S_{r}$). These text-aware scene embeddings ($S_{r}$) are
then concatenated with the visual embeddings ($V$) and the textual embeddings
($T$) and passed through a self-attention-based transformer module. Finally,
features are average-pooled and mapped to a predefined answer set to generate
the output answer.

####  2.2.2 Scene-embedded interaction Module

Fig. 4: Scene-embedded interaction module. It is a stack of layers of cross-
attention and self-attention. The cross-attention modulates the scene
embeddings based on the text queries, while the self-attention refines the
scene embeddings .

In SSG-QA-Net, initial scene embeddings $S$ capture global surgical scene
semantics. To handle complex questions that require localized focus, we
introduce a lightweight Scene-embedded Interaction Module (SIM). The main
objective of SIM is to correlate the textual embeddings with the scene
embeddings. SIM consists of two interaction layers. Each layer comprises self-
attention, cross-attention, and feed-forward sub-layers, as shown in Fig. 4.
The attention mechanism is defined as:

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V,$
(1)

where the Q is the short for query, K is key, and V is value.

In SIM, we first apply cross-attention, $S_{r}=\text{Cross-Attention}(S,T,T)$,
to the textual and scene embeddings by processing textual embeddings $T$ as
key and value, and scene embeddings $S$ as the query. This results in refined
scene embeddings, which incorporate textual cues. Then, the refined scene
embeddings are passed to the self-attention layer, $S_{r}=\text{Self-
Attention}(S_{r},S_{r},S_{r})$, to interact with themselves. By interacting
with the textual embeddings and the scene embeddings, we obtain the textual-
aware scene embeddings $S_{r}$. Through our ablation experiments, we show that
the $S_{r}$ significantly contributes to providing correct answers to fine-
grained questions.

##  3 Results and discussions

Table 1: We show that the SSG-QA dataset is more challenging and balanced than the EndoVis-18-VQA [20] and Cholec80-VQA [20]. SSG-QA dataset includes more attributes and complexities in the questions. Dataset | EndoVis-18-VQA [20] | Cholec80-VQA [20] | SSG-QA (Ours)
---|---|---|---
$\\#$Surgical Scenes | 2k | 21k | 25k
$\\#$Questions | 11k | 43k | 960k
$\\#$Unique Questions | 17 | 51 | 501k
Average Length in words | 5.8 | 2.0 | 12.8
Average $\\#$Questions per scene | 5.0 | 6.5 | 38.9

###  3.1 Dataset comparison

SSG-QA dataset contains the same train and test set videos as CholecT45 [17]
dataset, which contains $40$ laparoscopic cholecystectomy videos for training
and $5$ videos for testing. Our SSG-QA dataset contains $960k$ questions from
$25k$ surgical scenes. Table 7 presents the comparison between SSG-QA and the
typical datasets from prior work, i.e., EndoVis-18-VQA and Cholec80-VQA from
[20], showing a $8\times$ and $22\times$ increase in number of questions,
respectively. Also, our SSG-QA dataset contains more diverse questions per
scene ($38.9$ v.s. $6.5$) and much longer questions ($12.8$ words v.s. $5.8$
words). Furthermore, SSG-QA contains a wider range of categories for object
attributes, names, and inter-object relationships, as shown in the appendix.
Also, compared to the Cholec80-VQA dataset which provides $51$ questions for
all surgical scenes, our SSG-QA has more diverse questions ($501k$) that are
unique to surgical scenes, which prevents the model from overfitting to
specific question patterns.

###  3.2 Question-condition bias

Table 2: Identification of question-condition bias in existing datasets. We use the Accuracy, Recall, and Fscore metrics from SurgicalVQA [20]. Endovis-VQA contains significant question-condition bias because the language model with pure language inputs can outperform the model with vision and language inputs. Methods | Endovis-VQA [20] | Cholec80-VQA [20] | SSGQA
---|---|---|---
| Acc | Recall | Fscore | Acc | Recall | Fscore | Acc | Recall | Fscore
L+Bert [5] | 57.5 | 45.9 | 36.3 | 83.3 | 29.3 | 24.4 | 51.7 | 36.2 | 50.0
L+SciBert [2] | 55.8 | 45.1 | 36.9 | 83.3 | 29.3 | 24.4 | 52.4 | 36.9 | 49.4
L+ClinicalBert [9] | 60.4 | 50.7 | 40.8 | 83.1 | 29.4 | 24.4 | 52.3 | 35.4 | 49.9
VisualBert [15] | 61.9 | 41.2 | 33.4 | 89.7 | 62.9 | 63.3 | 55.0 | 42.5 | 54.8
VisualBert Resmlp [15] | 63.2 | 39.6 | 33.6 | 89.8 | 62.7 | 63.4 | 58.7 | 44.8 | 57.7

VQA systems can exploit the question-condition bias from the dataset as a
shortcut to answer questions without understanding the visual scenes. To
quantify this bias, we train language-only models like ClinicalBert [9] to
answer the questions without any visual information on existing VQA datasets,
such as EndoVis-18-VQA and Cholec80-VQA, and on our SSG-QA dataset. As shown
in Table 2, the language-only model ClinicalBert outperforms the vision-
language multi-modal models in EndoVis-18-VQA, suggesting the questions from
EndoVis-18-VQA contain simple shortcuts to the correct answer. Cholec80-VQA
and SSG-QA have a lower bias as their questions are more vision-relevant. SSG-
QA further reduces bias by using scene-graph-based diverse questions. In the
following, we perform the experiments on the Cholec80-VQA and SSG-QA dataset
due to their low question-condition bias.

Table 3: Comparison results for baselines and our models. The SSG-QA-Net with scene knowledge achieves the best results. The SSG-QA-Net (oracle) model refers to the model that uses detection labels from the SSG-QA dataset to construct the scene embeddings instead of using the trained YOLOv7 object detector. Models | Accuracy | mAP | Recall | Fscore
---|---|---|---|---
L+ClinicalBert | 52.3 | 40.4 | 35.4 | 49.9
VisualBert [15] | 55.0 | 47.9 | 42.5 | 54.8
VisualBert Resmlp [20] | 58.7 | 51.8 | 44.8 | 57.7
SSG-QA-Net | 60.7 | 54.9 | 49.1 | 60.3
SSG-QA-Net (Oracle) |  62.8 |  56.3 |  50.6 |  62.3
Table 4: Breakdown results of the prior models and our models. We show that our model outperforms the baselines by a large margin, especially on the complex questions that require visual reasoning. Also, the results on a different set of questions shows that our dataset is not dominated by one type of question. We report the mAP here. | VisualBERT [15] | VisualBERTMLP [20] | SSG-QA-Net | SSG-QA-Net (oracle)
---|---|---|---|---
Query Object | 39.4 | 38.4 | 48.0 | 55.4
Query Attribute | 51.7 | 54.5 | 54.8 | 60.2
Existence | 68.0 | 76.4 | 73.9 | 72.7
Counting | 24.5 | 29.6 | 36.9 | 24.2
Zero-hop | 50.4 | 53.2 | 56.6 | 55.0
One-hop | 46.4 | 46.2 | 50.3 | 51.9
Single-and | 23.4 | 30.9 | 39.0 | 41.4

###  3.3 Results of SSG-QA-Net

####  3.3.1 Results on SSG-QA

Comparison to other works. As shown in Table 3, SSG-QA-Net outperforms
baseline models like VisualBert [15] and VisualBert Resmlp [20] in metrics
such as mAP, Recall, and Fscore. We also train an upper-bound model, called
SSG-QA-Net (oracle), that uses the scene embeddings from detection labels of
the SSG-QA dataset instead of using the trained YOLOv7 object detector. This
model outperforms prior works significantly, emphasizing the importance of
high-quality scene embedding inputs.

Analysis by question type. As shown in Table 4, SSG-QA-Net can handle various
question types. For “counting” questions, it outperforms VisualBert by $7.3$
points in mAP. For “existence” and “query object” types, the model again shows
superior performance w.r.t to baseline models.

Analysis by complexity. SSG-QA dataset provides the diagnostic setup to
pinpoint the weakness of the model. As shown in Table 4, we compute the
performance of our models on questions that require different visual reasoning
complexity, i.e., zero-hop and one-hop, and single-and. Our model shows
consistent gains in both simple and complex question queries. For one-hop and
single-and type of questions, SSG-QA-Net achieves a $4.1$ and $8.1$ point mAP
increase over VisualBERT ResMLP, respectively. This indicates that the
inclusion of scene context can aid in resolving complex queries.

####  3.3.2 Results on Cholec80-VQA

Table 5: Results on the Cholec80-VQA. SSG-QA-Net achieves better results than the state-of-the-art models, even w.r.t SurgicalGPT, which contains a heavy sequence decoding module of GPT-2 | #Parameter | Accuracy | Recall | Fscore
---|---|---|---|---
VisualBert [15] | 209.8M | 89.7 | 62.9 | 63.3
VisualBert Resmlp [20] | 184.7M | 89.8 | 62.7 | 63.4
SurgicalGPT [19] | 234.5M | 87.5 | 57.5 | 57.9
SSG-QA-Net | 145.3M |  90.6 |  64.4 |  63.7

We also conduct the experiments on the other publicly available surgical VQA
dataset Cholec80-VQA. As illustrated in Table 5, SSG-QA-Net significantly
outperforms the SurgicalGPT [19], which requires heavy sequence decoding using
GPT-2 architecture. This highlights that the bottleneck of the current
surgical VQA problem lies in the visual scene understanding instead of text
generation. Also, even using YOLOv7 for object detection, our model achieves
higher performance with fewer parameters than prior works, verifying its
efficiency while maintaining higher performance metrics.

####  3.3.3 Ablation study

Table 6: Effect of different modules. RoIAlign pooling boosts results, and the Scene-embedded Interaction Module further enhances them. Both modules offer complementary benefits. SIM | ROI | Accuracy | mAP | Recall | Fscore
---|---|---|---|---|---
$\times$ |  $\times$ | 55.0 | 47.9 | 42.5 | 54.8
$\checkmark$ |  $\times$ | 57.1 | 51.0 | 47.2 | 56.1
$\times$ |  $\checkmark$ | 58.8 | 52.5 | 47.9 | 57.7
$\checkmark$ |  $\checkmark$ |  60.7 |  54.9 |  49.1 |  60.3

Table 6 shows that combining both the Scene-embedded Interaction Module (SIM)
and RoIAlign (ROI) pooling significantly boosts the model’s performance. This
suggests that these modules are not just individually beneficial but are
actually complementary. Specifically, the model attains the highest mAP
(54.9%) when both components are added. Also, the improvement indicates that
introducing scene knowledge representation learning is crucial for robust
surgical visual question answering.

##  4 Conclusion

In this paper, we tackle the problem of visual question answering (VQA) in the
context of fine-grained surgical scene understanding. First, we introduce a
new dataset called SSG-QA, which uses a surgical scene graph as an underlying
representation and a question-answer generation engine to generate diverse,
geometrically grounded, and surgical-action-oriented question-answer pairs.
The question-answer pairs are also sampled to mitigate the question-condition
bias that exists in the current surgical VQA datasets. We also propose a novel
model called SSG-QA-Net to explicitly incorporate scene knowledge and object-
wise local features in the VQA model design to improve the reasoning ability
on complex questions. The results show that SSG-QA-Net outperforms existing
baseline models by a large margin.

##  5 Acknowledgments

This work was supported by French state funds managed by the ANR within the
National AI Chair program under Grant ANR-20-CHIA-0029-01 (ChairAI4ORSafety)
and within the Investments for the future program under Grant ANR-10-IAHU-02
(IHU Strasbourg). This work was granted access to the HPC resources of IDRIS
under the allocations AD011013704R1, AD011011631R2, and AD011011631R3 made by
GENCI.

## References

  * Antol et al. [2015] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D., 2015.  Vqa: Visual question answering, in: ICCV, pp. 2425–2433.
  * Beltagy et al. [2019] Beltagy, I., Lo, K., Cohan, A., 2019.  Scibert: A pretrained language model for scientific text.  arXiv preprint arXiv:1903.10676 .
  * Carstens et al. [2023] Carstens, M., Rinner, F.M., Bodenstedt, S., Jenke, A.C., Weitz, J., Distler, M., Speidel, S., Kolbinger, F.R., 2023\.  The dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science.  Scientific Data 10, 1–8.
  * Chen et al. [2018] Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H., 2018.  Encoder-decoder with atrous separable convolution for semantic image segmentation, in: ECCV.
  * Devlin et al. [2018] Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805 .
  * He et al. [2017] He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017\.  Mask r-cnn, in: ICCV, pp. 2961–2969.
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: CVPR, pp. 770–778.
  * Hong et al. [2020] Hong, W.Y., Kao, C.L., Kuo, Y.H., Wang, J.R., Chang, W.L., Shih, C.S., 2020\.  Cholecseg8k: A semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80.  URL: https://arxiv.org/abs/2012.12453.
  * Huang et al. [2019] Huang, K., Altosaar, J., Ranganath, R., 2019.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.  arXiv preprint arXiv:1904.05342 .
  * Hudson and Manning [2019] Hudson, D.A., Manning, C.D., 2019\.  Gqa: A new dataset for real-world visual reasoning and compositional question answering, in: CVPR, pp. 6700–6709.
  * Jin et al. [2018] Jin, A., Yeung, S., Jopling, J., Krause, J., Azagury, D., Milstein, A., Fei-Fei, L., 2018.  Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks.  IEEE WACV .
  * Jocher [2020] Jocher, G., 2020.  YOLOv5 by Ultralytics.  URL: https://github.com/ultralytics/yolov5, doi:10.5281/zenodo.3908559.
  * Johnson et al. [2017] Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C.L., Girshick, R., 2017\.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, in: CVPR.
  * Krishna et al. [2016] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., Bernstein, M., Fei-Fei, L., 2016\.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  URL: https://arxiv.org/abs/1602.07332.
  * Li et al. [2019] Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W., 2019.  Visualbert: A simple and performant baseline for vision and language.  arXiv:1908.03557 .
  * Maier-Hein et al. [2022] Maier-Hein, L., Eisenmann, M., Sarikaya, D., März, K., Collins, T., Malpani, A., Fallert, J., Feussner, H., Giannarou, S., Mascagni, P., et al., 2022\.  Surgical data science–from concepts toward clinical translation.  MedIA 76, 102306\.
  * Nwoye et al. [2022] Nwoye, C.I., Yu, T., Gonzalez, C., Seeliger, B., Mascagni, P., Mutter, D., Marescaux, J., Padoy, N., 2022\.  Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos.  MedIA 78, 102433\.
  * Padoy [2019] Padoy, N., 2019.  Machine and deep learning for workflow recognition during surgery.  Minimally Invasive Therapy & Allied Technologies 28, 82–90.
  * Seenivasan et al. [2023] Seenivasan, L., Islam, M., Kannan, G., Ren, H., 2023\.  Surgicalgpt: End-to-end language-vision gpt for visual question answering in surgery.  arXiv preprint arXiv:2304.09974 .
  * Seenivasan et al. [2022] Seenivasan, L., Islam, M., Krishna, A.K., Ren, H., 2022\.  Surgical-vqa: Visual question answering in surgical scenes using transformer, in: MICCAI, pp. 33–43.
  * Twinanda et al. [2016] Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N., 2016\.  Endonet: a deep architecture for recognition tasks on laparoscopic videos.  IEEE transactions on medical imaging 36, 86–97.
  * Wang et al. [2023] Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M., 2023.  Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors, in: CVPR, pp. 7464–7475.

Advancing Surgical VQA with Scene Graph Knowledge
Supplementary Material

###  5.1 SSG Dataset

In this section, we first illustrate the statistics of the SSG-QA dataset and
show that it is more challenging and diverse than the prior works. Then, we
show the process of assigning location attributes to the surgical objects.
Finally, we visualize some samples of the scene graphs that we use to generate
question-answer pairs.

Table 7: Dataset statistics comparison. SSG-QA is more challenging and
balanced as it includes more attributes and complexities in the questions.

Dataset | EndoVis-18-VQA | Cholec80-VQA | SSG-QA
---|---|---|---
Average Length | 5.8 | 2.0 | 12.8
Average $\\#$Questions | 5.0 | 6.5 | 38.9
$\\#$Color | 0 | 0 | 7
$\\#$Type | 0 | 0 | 2
$\\#$Location | 4 | 0 | 6
$\\#$Object | 9 | 7 | 13
$\\#$Relation | 13 | 0 | 16

####  5.1.1 Dataset Statistics

As shown in Tab. 7, SSG-QA dataset contains the more challenging and diverse
question-answer pairs. Specifically, the average length of question is $38.9$
words compared to $5$ words and $6.5$ words from EndoVis-18-VQA and
Cholec80-VQA. Also, by using the question engine to automatically generate the
question-answer pairs, SSG-QA contains a more diverse and different pattern of
question-answer pairs for each surgical scene.

####  5.1.2 Location Attribute and Triplet Annotation

Fig. 5: SSG-QA Dataset - location attribute: (a) we use triplet annotations
for action-related question-answer generation; (b) we include a spatial
attribute for each object, e.g., top-right, bottom-left; (c) we use tool and
anatomy bounding boxes to generate question-answer pairs.

During the question-answer generation, we assign the location attributes to
the detected surgical objects, including instruments and anatomies. As shown
in Tab. 5, we split the image into $9$ spatial blocks and assign the location
attribute based on the locations of the surgical object in a given block. In
addition to the spatial relations among surgical objects, we also use triplet
annotation to connect the surgical objects with the actions, as shown in Fig.
5 (a).

####  5.1.3 Scene Graphs

Fig. 6: SSG-QA Dataset - scene graphs: The questions are generated by
exploiting the surgical scene graph. The surgical scene graph consists of
spatial attributes and action relations. Fig. 7: SSG-QA Dataset - different
complexities: We generate different type of question-answer pairs for each
surgical scene image. The text in red indicates the reasoning step in the
image, e.g., the fifth question requires two steps of reasoning to solve,
therefore categorized into single-and type. Fig. 8: Qualitative Results:
Comparison of SSG-QA-Net with baseline surgical VQA models on the question-
answer pairs based on spatial, color, counting, anatomy, and tool presence
attributes.

In this section, we show some scene graph samples that we use to generate
question-answer pairs. As shown in Fig. 6, the surgical semantic scene graphs
are directed graphs where the nodes represent the surgical objects and the
edges represent the spatial and action relation between nodes. Also, we assign
attributes to each surgical object, e.g., type, color, and location of the
objects. Note that the spatial relations are fully connected among the
objects. We show the important relations for a simpler demonstration.

####  5.1.4 Complexities of Question-answer Pairs

As the SSG-QA dataset is generated with a question engine and question
templates automatically, we control the complexity of question-answer pairs by
altering the question templates. The question-answer pairs with different
complexities also offer the diagnostic ability to pinpoint the weakness of the
VQA model. As shown in Fig. 7, we have $5$ kinds of question-answer pairs with
$3$ kinds of complexities. Here, the complexity is measured based on how many
reasoning steps are required to answer the question. For example, the fifth
question-answer in Fig. 7 requires two steps of reasoning to solve,
highlighted in red. As a result, this question is more complex than the
others.

Fig. 9: Qualitative Results: Comparison of SSG-QA-Net with baseline surgical
VQA models on the question-answer pairs based on surgical action triplets.
Fig. 10: Qualitative Results: It shows that the questions from our SSG-QA
dataset require visual reasoning to solve. Also, our SSG-QA-Net with scene
knowledge outperforms the prior baselines in terms of complex questions.

###  5.2 Qualitative Results

In this section, we compare our SSG-QA-Net to the state-of-the-art surgical
VQA models in terms of the question that require the understanding of
attributes (Sec. 5.2.1), action triplet occurrence (Sec. 5.2.2) and reasoning
steps (Sec. 5.2.3) to solve.

####  5.2.1 Attributes

As shown in Fig. 8, the SSG-QA-Net model outperforms the prior works when the
questions contain attributes, such as location and color. For example, the
second subfigure in Fig. 8 shows that the SSG-QA-Net model understands the
location attributes such as “top-mid” and “bottom-mid” to resolve the
question. Compared to the other methods, it contains more detailed knowledge
about the surgical scene.

####  5.2.2 Action Triplets

We also compare the SSG-QA-Net model to the prior works on the questions that
require an understanding of surgical action triplet. Fig. 9 shows that the
SSG-QA-Net model can answer the questions that are generated based on the
triplet annotations.

####  5.2.3 Reasoning Steps

Fig. 10 shows that SSG-QA-Net can perform multiple steps of reasoning to
generate the correct answer to complex questions. Compared to the prior works,
it achieves better results when the questions are more complex.