#  CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in
Healthcare

Akash Ghosh1, Arkadeep Acharya1, Raghav Jain1, Sriparna Saha1, Aman
Chadha2,3111Work does not relate to position at Amazon.,
Setu Sinha4

###### Abstract

In the era of modern healthcare, swiftly generating medical question summaries
is crucial for informed and timely patient care. Despite the increasing
complexity and volume of medical data, existing studies have focused solely on
text-based summarization, neglecting the integration of visual information.
Recognizing the untapped potential of combining textual queries with visual
representations of medical conditions, we introduce the Multimodal Medical
Question Summarization (MMQS) Dataset. This dataset, a major contribution of
our work, pairs medical queries with visual aids, facilitating a richer and
more nuanced understanding of patient needs. We also propose a framework,
utilizing the power of Contrastive Language Image Pretraining(CLIP) and Large
Language Models(LLMs), consisting of four modules that identify medical
disorders, generate relevant context, filter medical concepts, and craft
visually aware summaries. Our comprehensive framework harnesses the power of
CLIP, a multimodal foundation model, and various general-purpose LLMs,
comprising four main modules: the medical disorder identification module, the
relevant context generation module, the context filtration module for
distilling relevant medical concepts and knowledge, and finally, a general-
purpose LLM to generate visually aware medical question summaries. Leveraging
our MMQS dataset, we showcase how visual cues from images enhance the
generation of medically nuanced summaries. This multimodal approach not only
enhances the decision-making process in healthcare but also fosters a more
nuanced understanding of patient queries, laying the groundwork for future
research in personalized and responsive medical care.
Disclaimer: The article features graphic medical imagery, a result of the
subject’s inherent requirements.

## Introduction

In the midst of the global COVID-19 pandemic, healthcare systems have been
inundated with an unprecedented volume of inquiries, concerns, and
uncertainties. Individuals worldwide are posing health-related inquiries on
online platforms, seeking clarity and guidance on symptoms, prevention,
treatments, and vaccinations. These questions often employ everyday language
and encompass both pertinent and extraneous details that may not directly
pertain to the sought-after solutions. The complexity and urgency of the
situation, coupled with a considerable imbalance in the doctor-to-patient
ratio across many countries, have made the ability to swiftly and
comprehensively comprehend a patient’s query paramount (Abacha and Demner-
Fushman 2019; Yadav et al. 2021). In this context, medical question
summarization has emerged as a vital tool to distill information from consumer
health questions, ensuring the provision of accurate and timely responses.
However, existing works have overlooked the untapped potential of integrating
visual data, particularly images, with textual information. The motivations
for focusing on visual aids within medical question summarization (MQS) are
manifold. A significant portion of the population lacks familiarity with
medical terms needed to accurately describe various symptoms, and some
symptoms are inherently challenging to articulate through text alone. Patients
may also be confused between closely related symptoms, such as distinguishing
between skin dryness and skin rash. The combination of text and images in
medical question summarization can offer enhanced accuracy and efficiency,
providing a richer context that textual analysis alone may miss. This approach
recognizes the complex nature of patient queries, where photographs of
symptoms, medical reports, or other visual aids could provide crucial
insights. By focusing on the integration of images, researchers and healthcare
providers can respond to the evolving challenges of modern healthcare
communication.

Figure 1: Broad categorization of medical disorders in the MMQS Dataset
(MMQSD). The number of data points corresponding to each category has been
provided under each category in the above figure.

Large Language Models (LLMs) (Kojima et al. 2023) and Vision Language Models
(VLMs) (Zhang et al. 2023) have exhibited remarkable capacities in generating
human-like text and multimedia content. This prowess has driven their
deployment across the medical domain, predominantly for domain-specific tasks
such as chest radiograph summarization (Thawkar et al. 2023) and COVID-19 CT
report generation (Liu et al. 2021). Yet, their application in multimodal
medical question summarization remains uncharted territory. Leveraging zero-
shot and few-shot learning capabilities of these models (Dong et al. 2022)
offers a compelling advantage, especially for tasks like multimodal medical
question summarization, characterized by inherent data scarcity. However,
while the potential of LLMs and VLMs in this domain is undeniable, they aren’t
without constraints. Predominantly, generic LLMs and VLMs often lack a solid
grounding in task-specific knowledge, risking the generation of summaries that
might miss intricate details like symptoms, diagnostic tests, and medical
complexities. On the visual front, even as VLMs have excelled in typical
visual-linguistic tasks, medical imaging presents unique challenges. Efforts
like SkinGPT4 (Zhou et al. 2023), which fine-tunes MiniGPT4 (Zhu et al. 2023)
on skin disease images and clinical notes, are still notably domain-specific.
Medical images are inherently complex, demanding a profound understanding of
medical terminology and visual conventions, often necessitating an expert
medical practitioner’s perspective for accurate interpretation. This
complexity, combined with potential gaps in contextual understanding, can
result in models producing misleading or irrelevant summaries.

To address the limitations of LLMs and VLMs in multimodal medical question
summarization, we’ve conceived the CLIPSyntel framework. The first stage, the
Medical Disorder Identification Module, combines Large Language Model(LLM)
with Contrastive Language Image Pretraining(CLIP) (Radford et al. 2021),
forming a novel zero-shot classification approach to identify disorders from
visual images, utilizing the unique benefits of zero or few-shot learning.
Subsequently, during the Context Generation Phase, LLM adds context to the
medical query, focusing on vital components like symptoms, tests, and
procedures, mindful of the potential pitfalls of extraneous content.
Addressing this challenge, the Context Filtration Module filters the content
using a multimodal knowledge selection technique, preserving only the most
relevant information. The Summary Generation Phase follows, where an LLM
crafts the final medically accurate summaries based on the distilled
knowledge. Finally, to validate CLIPSyntel, we created the MMQS dataset, rich
in visual and textual representations of patients’ symptoms and
queries222https://github.com/AkashGhosh/CLIPSyntel-AAAI2024. This careful,
step-by-step structuring of CLIPSyntel allows it to bridge the divide between
general-purpose models and the niche requirements of medical question
summarization, effectively integrating textual and visual data to create
precise and context-aware medical summaries. To summarize we make the
following main contributions:
A novel task of Multimodal Medical Question Summarization for generating
medically nuanced summaries.
A novel dataset, MMQS Dataset, to advance the research in this area.
A novel metric MMFCM to quantify how well the model captures the multimodal
information in the generated summary.
A novel framework, ClipSyntel that harnesses the power of CLIP and LLMs to
augment the patient question with an additional rich context from visual
symptoms for the generation of final summaries.

(a) Distribution of Question Lengths.

(b) Distribution of Summary Lengths.

Figure 2: Distribution of question lengths and their summaries in the MMQS
dataset;

## Related Works

Medical Question Summarization: The task of Medical Question Summarization
(MQS) was initially introduced in 2019 with the development of the MeQSum
dataset (Abacha and Demner-Fushman 2019) specifically for this purpose. Early
research on MQS employed vanilla seq2seq models and pointer generator networks
to generate summaries. In 2021, a contest was held focusing on the generation
of medical domain summaries (Abacha et al. 2021). Participants utilized
various pre-trained models such as PEGASUS (Zhang et al. 2020), ProphetNet (Qi
et al. 2020), and BART (Lewis et al. 2019). Techniques like multi-task
learning were employed, utilizing BART to jointly optimize question
summarization and entailment tasks (Mrini et al. 2021). Another approach
involved reinforcement learning with question-aware semantic rewards derived
from two subtasks: question focus recognition (QFR) and question type
identification (QTR) (Yadav et al. 2021).

Role of Multimodality: In order to receive accurate treatment and guidance
from a medical expert, our responsibility is to effectively and efficiently
communicate the medical symptoms which can be done with additional visual
cues. Previously, many studies also showed that adding multimodality through
visual cues improves the performance of various medical tasks. Tiwari et al.
(2022) shows how multimodal information helps in building better Disease
Diagnosis Virtual Assistants. Delbrouck, Zhang, and Rubin (2021) also showed
how incorporating images helps in better summarization of radiology reports.
Gupta, Attal, and Demner-Fushman (2022) also showed the benefit of
incorporating videos for medical QA task The current work is motivated by this
idea of how integrating the patient’s provision of an image of their medical
condition alongside the text, improves the generation of more medically rich
summaries. We curate a multimodal dataset based on an existing MQS dataset,
focusing on a predefined set of symptoms, and propose a novel framework that
combines cutting-edge Multimodal Foundation Models like CLIP with Large
Language Models (LLMs). To our best understanding, this work is the first to
tackle the task of question summarization in the medical domain, particularly
within a multimodal setting.

## MMQS (Multimodal Medical Question Summarization) Dataset

To the best of our knowledge, a freely available multimodal question
summarization dataset that includes both textual questions and corresponding
medical images of patients’ conditions does not currently exist. To construct
such a dataset, we utilized the preexisting HealthCareMagic Dataset, which is
derived from the MedDialog data introduced by (Mrini et al. 2021). The initial
dataset comprised 226,395 samples, of which 523 were found to be duplicates.
To ensure data integrity and fairness, we eliminated these duplicate entries.
In order to ascertain the medical symptoms or signs that could effectively be
conveyed through visual means, we consulted a medical professional who also
happens to be a co-author of this paper. Following a series of brainstorming
sessions and carefully analyzing the dataset, we identified 18 symptoms that
are hard to specify only through text. These 18 symptoms are broadly divided
into four broad groups of multimodal symptoms to incorporate into our dataset.
These groups (refer to figure 1) are categorized as ENT (Ear, Nose, and
Throat), EYE (Eye-related), LIMB (Limbs-related), and SKIN (Skin-related).
From the ENT category, we selected symptoms including lip swelling, mouth
ulcers, and swollen tonsils. From the EYE category, we choose swollen eyes,
eye redness, and itchy eyelids. For the LIMB category, we included symptoms
such as edema, foot swelling, knee swelling, hand lumps, and neck swelling.
Lastly, from the SKIN category, we included symptoms of skin rash, skin
irritation, and skin growth. To extract images for the corresponding symptoms,
Bing Image Search API333https://www.microsoft.com/en-us/bing/apis/bing-image-
search-api was used. Then the images extracted are verified by a group of
medical students who are led by a medical expert. Therefore, in curating our
ultimate multimodal dataset, we selectively included instances from the
HealthCareMagic dataset that featured textual mentions of body parts such as
skin, eyes, ears, and others, both within the questions and their
corresponding summaries. We refrained from directly searching for symptom
names, as our intention was to encompass instances from the dataset where
uncertainty regarding the medical condition exists. Consequently, patients
implicitly allude to symptoms rather than explicitly stating their names. To
identify these relevant samples, we employed the Python library
FlashText444https://pypi.org/project/flashtext/1.0/ for efficient term
matching in the aforementioned dataset. The Python library Textblob4 was used
to correct the spelling of many improperly spelled words. To address the
misspelling of numerous words, the
Textblob4555https://pypi.org/project/textblob/0.9.0/ Python library was
employed. Following an initial exploration using FlashText, we initially
collected approximately 5000 samples. After meticulous manual validation of
each sample, our refinement process led us to a final dataset comprising 3015
samples where multimodality could be incorporated for the final dataset
creation.

(a) Sample instances from the MMQS-dataset.

1.192

(b) Medical Disorder Identification Module

Figure 3: Comprehensive illustration of Sample Instances from the MMQS-dataset
and Medical Disorder Identification Module. Figure 4: Flowchart depicting the
workflow of the proposed model CLIPSyntel. The diagram illustrates the step-
by-step process of ClipSyntel’s functioning.

### Data Annotation

We extracted 100 random samples from the filtered dataset, each comprising a
patient’s inquiry alongside its corresponding summary, and gave them to the
medical expert for annotation. The methodology through which the medical
expert annotated these samples is explained with the given example. For
example, suppose the patient is complaining regarding something that has
happened near their tonsils, but they are not able to name the exact medical
disorder they are suffering from. But their textual reference gave enough
context to understand that the patient is suffering from the disorder named
swollen-tonsils. After understanding this, the medical expert adds statements
like Please see what happened to my tonsils in the image below in the context
and accordingly adds a visual image representation of the disorder. In this
way, visual medical signs are incorporated into the textual medical question.
Additionally, the medical expert deduced that the conventional golden
summaries did not align well with the multimodal queries, prompting the need
for corresponding updates. To address these issues, the medical expert himself
modified these 100 samples by revising the questions, incorporating multimodal
information, and adapting the golden summaries accordingly. Subsequently, we
engaged with medical students and graduates to annotate the remaining samples
based on the guidelines given by the medical expert666The students were
compensated through gift vouchers and honorarium.. To quantify the level of
agreement among annotators, we calculated the kappa coefficient (k), which
yielded a value of 0.78. This coefficient indicates a noteworthy level of
consistent annotation. Following this rigorous process, the Multimodal Medical
Question Summarization (MMQS) dataset comprised a total of 3,015 samples. The
median length of the question in the final dataset is 442 words and the median
length of the question summary is 121 words, and example data points are also
shown in Table-3(a). The distributions of question length and question summary
length are shown in Figure 2(a) and Figure 2(b), respectively.

## Methodology

Problem Formulation: Each data point consists of a patient query in textual
form $T$ and an image $I$ of the medical disorder that the patient is
suffering from or trying to address in his textual query. The final output is
a concise summary $S$ of the query which incorporates information from both
the modalities. This section presents the novel architecture of CLIPSyntel
shown in Figure-4, a multimodal, knowledge-grounded framework designed to
generate medically nuanced summaries.For a comprehensive understanding, we
partition our approach into four distinct modules: (i) Medical Disorder
Identification Module, (ii) Contextual Information Generation Module, (iii)
Context Filtration Module, and (iv) Summary Generation Module

### Medical Disorder Identification

For the process of generating pertinent medical knowledge it is necessary to
accurately identifying the medical disorder that the patient is experiencing.
To achieve this, we employed the following structured approach: (1)
Utilization of CLIP: CLIP, designed to establish connections between images
and text, is initially applied. Despite its applicability to various visual
classification tasks, CLIP’s effectiveness in our Medical Disorder
Identification Task using only the names of the disorder is limited due to the
complex nature of medical images.  (2) Enhanced Contextualization through
Prompts: To provide better context for a particular medical disorder, we
prompted GPT 3.5 using the prompt: Describe the image of the disease
$\\{DISEASE\\}$ in a single sentence stating all the necessary details. This
approach yields improved contextual information for the specific medical
condition. (3) Identification of Probable Disorders: When presented with an
image of a medical disorder, we provided the contextual information for all 18
medical disorders, along with the corresponding image, to the CLIP model.
Subsequently, we selected the top 3 most likely medical disorders based on
CLIP’s analysis. (4) Final Prediction with GPT-3.5: These three probable
diseases, along with the medical query, is then subsequently passed to
LLM(GPT-3.5) using the prompt which is shown in Figure-3(b) to predict the
final medical disorder. Considering only the most probable disorder prediction
from CLIP yields an accuracy of 84 %. However, when incorporating the top 3
most probable disorders alongside the context after passing through the LLM,
the accuracy is enhanced to 87% in a zero-shot setup. The entire pipeline for
this module, displaying its logical flow and connections, is depicted in
Figure 3(b).

### Contextual Medical Knowledge Generation

In cases where patients may lack awareness of their medical condition, and
textual inquiries are insufficient, acquiring additional knowledge about their
specific symptoms and the necessary tests and procedures (as advised by
medical experts) proves essential for creating a meaningful summary,
especially in a multimodal setup. Formally, after identification of the
medical disorder $M$, we formulate a set of prompts
$P=\\{p_{0},p_{1},\ldots,p_{n}\\}$ with the aim of generating diverse
contextual information about that disorder $M$,
$KS=\\{ks_{0},ks_{1},\ldots,ks_{n}\\}$ by promting GPT-3.5 as follows. The
Prompts used to generate contextual information are as below:

(1) What are the symptoms of the medical condition ⟨medical disorder ⟩?

(2) What tests and procedures need to be done for the medical condition
⟨medical disorder ⟩?

| Model | ROUGE | BLEU | BERTScore
---|---|---|---|---
| R1 | R2 | RL | B1 | B2 | B3 | B4
LLAMA-2 | LLM(only texual query) | 0.188 | 0.051 | 0.159 | 0.144 | 0.064 | 0.032 | 0.016 | 0.822
Clip+GPT-3.5 | 0.189 | 0.052 | 0.161 | 0.156 | 0.070 | 0.036 | 0.018 | 0.819
| CLIPSyntel | 0.249 | 0.081 | 0.211 | 0.195 | 0.102 | 0.059 | 0.034 | 0.864
RedPajama | LLM(only texual query) | 0.146 | 0.023 | 0.125 | 0.086 | 0.030 | 0.011 | 0.004 | 0.828
Clip+GPT-3.5 | 0.148 | 0.025 | 0.132 | 0.087 | 0.0314 | 0.012 | 0.004 | 0.831
| CLIPSyntel | 0.157 | 0.0279 | 0.140 | 0.088 | 0.032 | 0.014 | 0.006 | 0.838
Vicuna | LLM(only textual query) | 0.372 | 0.160 | 0.318 | 0.385 | 0.243 | 0.161 | 0.102 | 0.905
Clip+GPT-3.5 | 0.384 | 0.164 | 0.321 | 0.391 | 0.245 | 0.167 | 0.105 | 0.906
| CLIPSyntel | 0.391 | 0.167 | 0.33 | 0.40 | 0.25 | 0.171 | 0.108 | 0.910
FLAN-T5 | LLM(only textual query) | 0.220 | 0.042 | 0.205 | 0.129 | 0.052 | 0.026 | 0.012 | 0.88
Clip+ GPT-3.5 | 0.220 | 0.042 | 0.205 | 0.136 | 0.056 | 0.026 | 0.012 | 0.88
| CLIPSyntel | 0.243 | 0.068 | 0.215 | 0.182 | 0.096 | 0.052 | 0.0256 | 0.891
GPT-3.5 | LLM(only textual query) | 0.451 | 0.227 | 0.396 | 0.471 | 0.330 | 0.243 | 0.170 | 0.920
Clip + GPT-3.5 | 0.452 | 0.226 | 0.399 | 0.471 | 0.331 | 0.242 | 0.171 | 0.920
| CLIPSyntel | 0.463 | 0.241 | 0.409 | 0.478 | 0.342 | 0.257 | 0.186 | 0.921

Table 1: Performance of various CLIPSyntel models and corresponding baselines,
evaluated using automatic metrics with different LLMs.

### Multimodal Medical Knowledge Filtration

While we anticipate that GPT-3.5 will generate useful and pertinent
information based on the given prompt, it occasionally engages in
hallucinations and produces irrelevant content in relation to the primary
inquiry. This hallucinated text has the potential to divert Large Language
Models (LLMs) from generating high-quality summaries. It may also generate
content that is not pertinent for doctors to investigate the case swiftly.
Particularly in fields such as medicine, one must exercise heightened caution
to minimize misinformation, given that errors could have serious consequences
for patients. To tackle this issue, we present a filtering strategy called
Multimodal Medical Knowledge Filtration, which operates as follows. Formally,
for each $ks_{i}$, which corresponds to a specific text field in the $KS$, it
is further broken down into a set of $k$ sentences
$ks_{i}=\\{s_{1},s_{2},\ldots,s_{m}\\}$ using sentence
tokenization777https://www.nltk.org/api/nltk.tokenize.html. To achieve this,
we employ a pre-trained multimodal model, ImageBind (Girdhar et al. 2023),
which serves as an off-the-shelf encoder-based multimodal model $enc(\cdot)$
that maps a token sequence (text) and an image to their respective feature
vectors embedded in a unified vector space. Cosine similarity
$sim(\cdot,\cdot)$ is then used to measure the relevance of each sentence to
the image. Formally, a sentence $s_{j}$ is retained if
$sim(enc(s_{j}),enc(I))>Th$, where $I$ is the image associated with the
medical disorder, and $Th$ is the predefined similarity threshold. Building
upon this, we define the subset of retained knowledge sentences, having
undergone the Multimodal Medical Knowledge Filtration module (MMKFS), as
follows:

$\displaystyle\leavevmode\resizebox{418.08165pt}{}{$Ms^{\prime}_{i}=\\{s_{j}\,|\,s_{j}\in
ks_{i}\,\text{and }\text{sim}(\text{enc}(s_{j}),\text{enc}(I))>Th\\}$}$ (1)

With this approach, the filtered knowledge sentences
$MS^{\prime}=\\{ms^{\prime}_{0},\ldots,ms^{\prime}_{n}\\}$ not only hold a
high degree of visual-textual alignment with the corresponding medical
disorder image but are also contextually relevant.

### Summary Generation Module

This component is crafted to leverage the refined knowledge sentences, denoted
as $MS^{\prime}$ in order to craft informative and contextually relevant
summaries for patients’ healthcare inquiries. The Summary Generation Module
functions with a pre-trained Large Language Model (LLM), referred to as $LM$.
For summary creation, $LM$ is provided a specially crafted prompt $P{in}$ that
integrates both the actual patient query $X$ and the generated medical
disorder knowledge $MS^{\prime}$. The outcome of this process is the generated
summary text, denoted as $S$. In a more formal representation, this process
can be articulated as: $S=LM(P{in}(X,MS^{\prime}))$. The prompt used to
generate the final summary is presented below:

Write a very short and concise one line summary of the following dialogue as a
question in a healthcare forum incorporating the relevant medical symptoms in
the additional context given below. Dialogue:⟨Patient Query ⟩and Additional
Context:⟨Filtered Medical Context⟩

| Model(GPT-3.5) | Clinical-EvalScore | Factual Recall | Omission Rate | MMFCM Score
---|---|---|---|---|---
| LLM(Patient Query) | 3.4 | 0.748 | 0.235 | 0.9
Clip + GPT-3.5 | 3.51 | 0.792 | 0.2215 | 1.52
CLIPSyntel | 3.62 | 0.818 | 0.2217 | 1.6
Annotated Summary | 4.1 | 0.889 | 0.144 | 1.86

Table 2: Human evaluation scores of the best CLIPSyntel model and their
corresponding baselines across different metrics.

## Experimental Results and Discussion

Experimental Setup: We leveraged the following general purpose LLMs for
summary generation module:
RedPajama888https://huggingface.co/togethercomputer/RedPajama-INCITE-
Chat-3B-v1, FLAN-T5 (Chung et al. 2022), Vicuna (Zheng et al. 2023) and
GPT-3.5. We tested these LLMs in different settings: With only the patient’s
question(text) in the prompt, With the patient’s question combined with the
knowledge obtained from (CLIP+GPT3.5), and then our proposed framework
((CLIPSyntel)). We have set the similarity threshold parameter $Th$ to 0.5 and
temperature to 0.5 across all settings based on a thorough investigation. We
utilize ROUGE (Lin 2004), BLEU (Papineni et al. 2002), and BERTScore (Zhang et
al. 2019) as automatic evaluation metrics. For the purpose of human
evaluation, we collaborate with a medical expert and a few medical students.
We have identified four distinct and medically nuanced metrics for this
evaluation: clinical evaluation score, factual recall (Abacha et al. 2023),
omission rate(Abacha et al. 2023), and our newly introduced MMFCM score.

Automated Evaluation: The results presented in Table 1 provide valuable
insights into the performance of various models, highlighting the
effectiveness of different approaches in the context of the task. Below, we
discuss some key observations and trends: (1) CLIPSyntel Performance: Across
all LLMs, CLIPSyntel consistently performed the best. This indicates the
robustness and versatility of CLIPSyntel, demonstrating its ability to
outshine other base models. The results suggest the capability of CLIPSyntel’s
design to effectively leverage both textual and visual information. These
results also show that adding contextual information does help in our task.
(2) GPT-3.5 based Models’ Superiority: Among the various models evaluated, the
GPT-3.5 based models stood out as the top performers compared to open-source
models. The high scores across multiple metrics reflect the sophisticated
design and capability of GPT-3.5. (3) Best Performance Among Open-Source LLMs:
Vicuna was the best performer among open-source LLMs. Its scores were notably
higher compared to other open-source counterparts, signaling its potential as
an effective alternative for specialized tasks.

Human Evaluation: The human evaluation was done by a team of medical students
led by a doctor. The team was given 10 % of the dataset (selected at random)
for evaluation purpose and was asked to rate the summaries generated, which
takes only patient question (text) as context, patient query in addition to
the medical disorder context (CLIP + GPT-3.5) and finally the summary
generated by our proposed pipeline (CLIPSyntel). The following metrics are
used for the evaluation: (1) Clinical Evaluation Score: The doctor and his
team were asked to rate the summaries between 1 (poor) and 5 (good) based on
their overall relevance, consistency, fluency, and coherence. (2) Multi-modal
fact capturing metric (MMFCM): We propose a new metric to evaluate how well a
model incorporates relevant medical facts and identifies the correct disorder
in a multimodal setup. MMFCM is calculated by considering both the facts
extracted from the medical query and image and assessing whether they are
correctly incorporated in the generated summary. The metric accounts for (1)
The ratio of correct facts in the summary to the total number of relevant
medical facts. (2) Additional scores based on the accuracy of the disorder’s
detection, with values ranging from +2 for fully correct to -1 for incorrect
identification. See Algorithm 1 for the detailed algorithm.

Algorithm 1 MMFCM Method

0:
$F_{m}=\\{\text{fact}_{m,1},\text{fact}_{m,2},\ldots,\text{fact}_{m,n-1},\text{fact}_{m,n}\\}$
{Relevant Medical facts from query ’m’.}

0: $Sf_{m}$ =$\\{{Summfact_{m,1},\dots,Summfact_{m,n}}\\}${ Relevant Medical
facts of summary of query ’m’.}

$\\#CorrectFacts_{m}=\lvert F_{m}\cap Sf_{m}\rvert$
{Number of correct medical facts in each summary}

if $\left(\text{Correct Medical Disorder phrase}\in\\{F_{m}\cap
Sf_{m}\\}\right)$ then

$\\#CorrectFacts_{m}+=2$

else if $\left(\text{Partially correct disorder phrase}\in\\{F_{m}\cap
Sf_{m}\\}\right)$ then

$\\#CorrectFacts_{m}+=1$

else if $\left(\text{Incorrect disorder phrase}\in\\{F_{m}\cap
Sf_{m}\\}\right)$ then

$\\#CorrectFacts_{m}+=-1$

else

$\\#CorrectFacts_{m}+=0$

end if

return MMFCM = $\frac{\\#\text{CorrectFacts}_{m}}{\lvert F_{m}\cap
Sf_{m}\rvert}$

(3) Medical Fact Based Metrics: We employed the Factual Recall and Omission
Recall metrics (Abacha et al. 2023) to assess the extent to which the
generated summary captures medical facts compared to the gold standard
annotated summary.

Table 2 offers a comprehensive comparative analysis, underscoring ClipSyntel’s
distinct advantages over various baseline methods across multiple human
evaluation performance metrics. Notably, our proposed architecture excels the
summaries that incorporate only the patient’s textual information,
highlighting the valuable impact of fusing visual symptom cues with textual
patient queries, resulting in more clinically nuanced summaries. Moreover, the
augmentation of medical knowledge is affirmed by substantial enhancements in
Factual Recall and Omission Rate metrics, further corroborated by our proposed
Multimodal fact Capturing Metric(MMFCM) score. Overall, ClipSyntel emerges as
a clear frontrunner in human evaluation metrics, outperforming competing
baselines by a considerable margin.

Figure 5: Sample Summaries generated by two variations GPT-3.5 model: only
question and CLIPSyntel

Qualitative analysis: The Figure 5 provides an insightful comparison of the
summaries generated by two variations of the GPT-3.5 model, namely the ”Only
Question” and ”CLIPSyntel” configurations, for medical inquiries. The analysis
mainly focuses on two aspects: (1) CLIPSyntel Captures Medical Disorders
Correctly: The CLIPSyntel variant demonstrates a noteworthy ability to
identify and represent medical conditions with precision. In both cases
presented in the table, CLIPSyntel has shown an understanding of the
underlying medical issues. (2) CLIPSyntel Does Less Hallucinations:
CLIPSyntel’s summaries stick to the information explicitly provided in the
medical questions. Unlike the ”Only Question” variant that might introduce
slight changes or uncertainties, CLIPSyntel maintains a strict adherence to
the facts. This ability to avoid unnecessary extrapolations or inaccuracies
enhances CLIPSyntel’s reliability.

## Risk Analysis

It’s important to note several limitations in our approach. We restricted our
study to a zero-shot prompting strategy, not fully examining how prompt
variations might affect results. Second, although CLIP performs well in low-
resource environments, we must be cautious of potential misclassifications of
medical images, as these could lead to serious or even fatal misinformation in
a summary generation. Despite the promising performance of CLIPSyntel in
various scenarios, the risk of misinformation in healthcare is significant.
Therefore, involving a medical expert is vital to ensure that our AI model
serves as an aid, not a replacement, for medical professionals.

## Conclusion and Future Work

In this study, we delve into the impact of incorporating multi-modal cues,
particularly visual information, on question summarization within the realm of
healthcare. We present the MMQS dataset, comprising 3015 multimodal medical
queries with golden summaries that merge visual and textual data. This novel
collection fosters new assessment techniques in healthcare question
summarization. We also introduce the CLIPSyntel framework, leveraging LLMs and
the CLIP model, to enhance summaries with visual symptom details. CLIP enables
symptom classification, and an ImageBind-filtering module mitigates content
hallucination. In our future endeavors, we aspire to develop a Vision-Language
model capable of extracting the intensity and duration details of symptoms and
integrating them into the patient query’s final summary generation.
Furthermore, our expansion plans encompass incorporating medical videos and
addressing scenarios involving code-mixed patient queries.

## Ethical Considerations

Summarization in healthcare necessitates strong ethical considerations,
particularly regarding safety, privacy, and potential bias. To address these
concerns in our project with the MMQS dataset, we implemented several
proactive measures. We collaborated closely with medical professionals and
also obtained IRB approval to ensure ethical rigor and patient privacy. We
rigorously followed legal and ethical guidelines 999https://www.wma.net/what-
we-do/medical-ethics/declaration-of-helsinki/ during dataset validation,
integration of images, and annotation of summaries. Medical experts were
engaged throughout the process, providing validation and correction of the
dataset and also validating the outputs of the models. The proposed dataset is
based on original HealthCareMagic Dataset; the medical questions/samples are
taken from this dataset. The incorporation of multimodality into the task is
done under the full supervision of a medical professional. Additionally, we
ensured user privacy by not disclosing identities.

## Acknowledgement

Akash Ghosh expresses his heartfelt gratitude to the Technology Innovation Hub
(TIH) IIT Patna and SERB for the awarded fellowship, a crucial source of
support that significantly bolsters his research endeavors.

## References

  * Abacha and Demner-Fushman (2019) Abacha, A. B.; and Demner-Fushman, D. 2019.  On the summarization of consumer health questions.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2228–2234.
  * Abacha et al. (2021) Abacha, A. B.; M’rabet, Y.; Zhang, Y.; Shivade, C.; Langlotz, C.; and Demner-Fushman, D. 2021.  Overview of the MEDIQA 2021 shared task on summarization in the medical domain.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , 74–85.
  * Abacha et al. (2023) Abacha, A. B.; Yim, W.-w.; Michalopoulos, G.; and Lin, T. 2023.  An Investigation of Evaluation Metrics for Automated Medical Note Generation.  _arXiv preprint arXiv:2305.17364_.
  * Chung et al. (2022) Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, E.; Wang, X.; Dehghani, M.; Brahma, S.; et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Delbrouck, Zhang, and Rubin (2021) Delbrouck, J.-B.; Zhang, C.; and Rubin, D. 2021.  QIAI at MEDIQA 2021: Multimodal Radiology Report Summarization.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , 285–290. Online: Association for Computational Linguistics.
  * Dong et al. (2022) Dong, Q.; Li, L.; Dai, D.; Zheng, C.; Wu, Z.; Chang, B.; Sun, X.; Xu, J.; and Sui, Z. 2022.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_.
  * Girdhar et al. (2023) Girdhar, R.; El-Nouby, A.; Liu, Z.; Singh, M.; Alwala, K. V.; Joulin, A.; and Misra, I. 2023.  Imagebind: One embedding space to bind them all.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 15180–15190.
  * Gupta, Attal, and Demner-Fushman (2022) Gupta, D.; Attal, K.; and Demner-Fushman, D. 2022.  A Dataset for Medical Instructional Video Classification and Question Answering.  arXiv:2201.12888.
  * Kojima et al. (2023) Kojima, T.; Gu, S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2023.  Large language models are zero-shot reasoners. arXiv.
  * Lewis et al. (2019) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Lin (2004) Lin, C.-Y. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , 74–81.
  * Liu et al. (2021) Liu, G.; Liao, Y.; Wang, F.; Zhang, B.; Zhang, L.; Liang, X.; Wan, X.; Li, S.; Li, Z.; Zhang, S.; et al. 2021.  Medical-vlbert: Medical visual language bert for covid-19 ct report generation with alternate learning.  _IEEE Transactions on Neural Networks and Learning Systems_ , 32(9): 3786–3797.
  * Mrini et al. (2021) Mrini, K.; Dernoncourt, F.; Chang, W.; Farcas, E.; and Nakashole, N. 2021.  Joint summarization-entailment optimization for consumer health question understanding.  In _Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations_ , 58–65.
  * Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 311–318.
  * Qi et al. (2020) Qi, W.; Yan, Y.; Gong, Y.; Liu, D.; Duan, N.; Chen, J.; Zhang, R.; and Zhou, M. 2020\.  Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training.  _arXiv preprint arXiv:2001.04063_.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 8748–8763. PMLR.
  * Thawkar et al. (2023) Thawkar, O.; Shaker, A.; Mullappilly, S. S.; Cholakkal, H.; Anwer, R. M.; Khan, S.; Laaksonen, J.; and Khan, F. S. 2023.  Xraygpt: Chest radiographs summarization using medical vision-language models.  _arXiv preprint arXiv:2306.07971_.
  * Tiwari et al. (2022) Tiwari, A.; Manthena, M.; Saha, S.; Bhattacharyya, P.; Dhar, M.; and Tiwari, S. 2022\.  Dr. can see: towards a multi-modal disease diagnosis virtual assistant.  In _Proceedings of the 31st ACM international conference on information & knowledge management_, 1935–1944.
  * Yadav et al. (2021) Yadav, S.; Gupta, D.; Abacha, A. B.; and Demner-Fushman, D. 2021.  Reinforcement learning for abstractive question summarization with question-aware semantic rewards.  _arXiv preprint arXiv:2107.00176_.
  * Zhang et al. (2023) Zhang, J.; Huang, J.; Jin, S.; and Lu, S. 2023.  Vision-language models for vision tasks: A survey.  _arXiv preprint arXiv:2304.00685_.
  * Zhang et al. (2020) Zhang, J.; Zhao, Y.; Saleh, M.; and Liu, P. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , 11328–11339. PMLR.
  * Zhang et al. (2019) Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2019.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_.
  * Zheng et al. (2023) Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023.  Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.  _arXiv preprint arXiv:2306.05685_.
  * Zhou et al. (2023) Zhou, J.; He, X.; Sun, L.; Xu, J.; Chen, X.; Chu, Y.; Zhou, L.; Liao, X.; Zhang, B.; and Gao, X. 2023.  SkinGPT-4: An Interactive Dermatology Diagnostic System with Visual Large Language Model.
  * Zhu et al. (2023) Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_.


figurec

# S-Nav: Semantic-Geometric Planning for Mobile Robots

Paul Kremer ID Hriday Bavle ID University of Luxembourg, Interdisciplinary
Center for Security Reliability and Trust (SnT), 29 Av. J.F. Kennedy, L-1855
Luxembourg, Luxembourg  Jose Luis Sanchez-Lopez ID University of Luxembourg,
Interdisciplinary Center for Security Reliability and Trust (SnT), 29 Av. J.F.
Kennedy, L-1855 Luxembourg, Luxembourg  Holger Voos ID University of
Luxembourg, Interdisciplinary Center for Security Reliability and Trust (SnT),
29 Av. J.F. Kennedy, L-1855 Luxembourg, Luxembourg  University of Luxembourg,
Faculty of Science, Technology and Medicine, 2 Avenue de l’Université, L-4365
Esch-sur-Alzette, Luxembourg

###### Abstract

Path planning is a basic capability of autonomous mobile robots. Former
approaches in path planning exploit only the given geometric information from
the environment without leveraging the inherent semantics within the
environment. The recently presented S-Graphs constructs 3D situational graphs
incorporating geometric, semantic, and relational aspects between the elements
to improve the overall scene understanding and the localization of the robot.
But these works do not exploit the underlying semantic graphs for improving
the path planning for mobile robots. To that aim, in this paper, we present
S-Nav a novel semantic-geometric path planner for mobile robots. It leverages
S-Graphs to enable fast and robust hierarchical high-level planning in complex
indoor environments. The hierarchical architecture of S-Nav adds a novel
semantic search on top of a traditional geometric planner as well as precise
map reconstruction from S-Graphs to improve planning speed, robustness, and
path quality. We demonstrate improved results of S-Nav in a synthetic
environment.

## License

For the purpose of Open Access, the author has applied a CC-BY-4.0 public
copyright license to any Author Accepted Manuscript version arising from this
submission.

##  1 Introduction

Mobile robots have gained a lot of traction in recent years and have seen
widespread use in different industries such as construction, mining, etc.,
where they are used for autonomous inspection tasks. To date, they are mostly
teleoperated or operated semi-autonomously under the supervision of a human
operator. Fully autonomous operation could thus significantly reduce costs,
however, several technical challenges such as perception, navigation, mapping,
and localization are currently detrimental to this mode of operation. Mobile
robots should not only create meaningful maps of the environment while
localizing within it but also be able to exploit these maps to perform fast
and efficient planning.

Traditionally, mobile robots build a geometric map [4], [5] of their
environment using simultaneous localization and mapping techniques (SLAM) in
combination with their onboard sensors [7] (e.g., LiDAR). Recently, we
presented S-Graphs a novel graph-based semantic SLAM that combines traditional
geometric SLAM with scene graphs [1], [2]. S-Graphs extracts the topological-
relational information of the environment such as wall surfaces, rooms, and
doorways including the topological connections between those semantic entities
enabling the robot to reason about its environment in a way humans would.
S-Graphs showed promising results in terms of precise robot localization and
high-level hierarchical map generation over a variety of datasets. However,
this scene knowledge is not yet leveraged for performing more intelligent and
faster path planning for mobile robots.

To bridge this gap, we leverage the metric, semantic, and relational
information in S-Graphs for the purpose of path planning. We propose a novel
hierarchical planner called S-Nav which leverages the semantic layer to
improve planning on the geometric layer. First, we perform a semantic graph
search utilizing the semantic elements within the S-Graphs to generate a
sparse undirected graph of semantic elements such as rooms and doorways. The
undirected global semantic graph is then divided into local subproblems which
can be solved in parallel and pose a set of simpler problems to the underlying
geometric planner. The main contributions of this work are:

  * •

Novel hierarchical planner called S-Nav utilizing geometric, semantic, and
relational information for faster planning.

  * •

Semantic planner for faster global plans.

  * •

Semantic subproblem solver further simplifies the global plan into local
subproblems for the underlying geometric planner.

A brief summary of S-Graphs is given in section 2.1. S-Nav, the novel
semantic-geometric planner, is introduced in section 3. The main blocks are
the Semantic Planner (section 3.1), the Subproblem Solver (section 3.2), and
the Geometric Planner (section 3.4). The evaluation and results are presented
in section 4. This work is concluded in section 5.

##  2 System Overview

The complete system architecture is shown in fig. 1. S-Nav builds on top of
S-Graphs and utilizes it as its main data source. S-Nav itself is composed of
the Semantic Planner, the Subproblem Solver, and the Geometric Planner. A path
query is first handled first by the Semantic Planner whose output serves as a
rough initial guess that cascades into the Geometric Planner via the
Subproblem Solver.

S-Graphs Semantic Planner Sampling-base Planner Global Map Geometric Planner
Graph Structure Semantic Planner Graph Data Query S-Nav Subproblem Solver
Robot Global Path Semantic-geometric Path Sensor Data Subproblem Subpath

Figure 1: Overview of S-Nav. A query (e.g., ’go from current position to the
kitchen’) is first handled by the Semantic Planner which provides an initial
high-level semantic-geometric path based on the graph structure obtained from
S-Graphs. Next, in the Subproblem Solver, the semantic-geometric path is
subdivided into smaller, easier problems that are individually solved by the
Geometric Planner. The individual paths between the subproblems are then
reassembled into the final high-level path that is passed to the robot.

###  2.1 Situational Graphs (S-Graphs)

S-Graphs is an optimizable graph structure built using online measurements
such as LiDAR data or markers [1], [2], [6].

The graph structure consists of five layers that are summarized as:

Keyframes Layer: Composed by the robot’s pose ${}^{M}x_{R_{i}}\in SE(3)$
constrained by the the robot’s odometry measurements.

Walls Layer: Each room is composed of four planes extracted from onboard
sensor measurements. They are constrained using pose-plane constraints.

Room Layer: A room is formed by its four planes constrained by a cost function
consisting of the room center ${}^{M}\mathbf{p}_{i}\in\mathbb{R}^{2}$ and
$w_{i}$ the distance between the opposite planar pairs.

Floor Layer: A floor is a collection of rooms optimized analogously to rooms
by extracting the largest distance between the opposite planar pairs.

Doorway Layer: A doorway marks the physical, traversable connection between
two rooms defined by a center point ${}^{M}\mathbf{d}_{i}\in\mathbb{R}^{2}$
and a width $r_{i}$, and is constrained by the physical distance between the
two rooms it connects.

S-Graphs serves as the main source of information for S-Nav. Therefore, this
work makes extensive use of the room and doorway layers. The presented
architecture can, however, easily be expanded to include multiple floors and
other semantic entities (e.g., objects).

##  3 S-Nav

S-Nav is our novel hierarchical semantic-geometric planning solution that
combines S-Graphs with an informed geometric planner. Our solution provides
the following benefits over traditional, purely geometric planners:

  * •

The geometric search can greatly profit from a rough initial guess provided by
the semantic layer by constraining the areas the planner can visit and by
providing subgoals toward the final goal.

  * •

A query in natural form, e.g., ’go from here to the kitchen’, can easily be
mapped to a semantic-geometric problem.

  * •

Handling forbidden areas such as closed doors or rooms that should not be
traversed is trivial on the semantic layer, whereas it would require map
changes on the geometric layer. Similarly, if a doorway is detected as
untraversable, replanning is very fast as the doorways node can easily be
disconnected from the graph.

Semantic-Graph LayerRoom ContoursGeometric LayerSigned Distance
FieldRoomsDoorwaysTopology

Figure 2: The semantic-graph layer is an undirected topological graph of rooms
and doorways extracted from features segmented by S-Graphs. The results of
this first layer cascade into the geometric layer which is formed by OMPL and
Voxblox. The contour layer reduces the valid state space of the geometric
planner to do informed decisions on where to sample. The semantic path is
highlighted in green together with the room contours that have to be traversed
along the path.

The structure of S-Nav is depicted in fig. 1 whilst the different layers are
visualized in fig. 2. Its main parts are formed by the Semantic Planner that
cascades into the Geometric Planner via the Subproblem Solver (SPS). The final
path is then passed to the robot, potentially via additional layers such as
local planning, trajectory generation, and motion control.

###  3.1 Semantic Search

The scene graph structure of S-Graphs encodes a high-level representation of
the environment the robot is operating in. Herein, this scene graph is
converted into an undirected graph connecting the semantic elements of the
scene. The connections (edges) between the elements have an associated cost,
i.e., for the room-to-doorway connections, a cost

$\displaystyle
c_{dr}=\lVert{}^{M}\mathbf{p}_{i}-{}^{M}\mathbf{d}_{i}\rVert^{2}+p_{d}$ (1)

is assigned, consisting of the distance between the center point of the room
and the associated doorway, plus a fixed penalty for doorway crossing. The
fixed penalty $p_{d}$ can be used to prefer a slightly longer path with fewer
(potentially closed) doorways.

Generally, the graph is sparse, featuring only a small number of nodes and
edges. As such, a shortest-path search using, e.g., A* is virtually free
compared to a full search on the geometric layer.

For a given query ’$\mathbf{p}_{s}\rightarrow\mathbf{p}_{g}$’ (read: from
$\mathbf{p}_{s}$ to $\mathbf{p}_{g}$), the semantic planner provides a
solution of type

$\displaystyle\begin{split}\mathbf{p}_{s}\rightarrow\mathbf{d}_{k}\rightarrow\mathbf{d}_{k+1}\rightarrow\cdots\rightarrow\mathbf{d}_{k+n}\rightarrow\mathbf{p}_{g}\\\
\textbf{via:}\hskip 2.84544ptF_{R}=R_{s}\cup R_{i}\cup\dots\cup R_{i+n}\cup
R_{g},\hskip 5.69046ptF_{R}\subseteq S\subset\mathbb{R}^{3}\end{split}$ (2)

where $\mathbf{p}_{s},\mathbf{p}_{g}$, $R_{s},R_{g}$ are the start and goal
positions and rooms, $\mathbf{d}_{k}$ the doorway center points to traverse
along the route, $R_{(.)}$ are the rooms, i.e., the free space the robot has
to pass through to reach its destination, $S$ is the state space limited by
the bounding box of the map, and $F_{R}$ is the reduced free space obtained
from the semantic planner that is passed to the geometric planner. By
restricting the geometric planner to $F_{R}$, its sampler can be much smarter
about the placement of the samples.

###  3.2 Semantic Search with Subproblems

The global problem in eq. 2 can further be simplified into a set of local
subproblems that can be solved in parallel and, individually, pose a simpler
problem to the geometric planner:

$\displaystyle 1:\mathbf{p}_{s}$ $\displaystyle\rightarrow\mathbf{d}_{k},$
$\displaystyle\phantom{0}\textbf{via: }R_{s\phantom{+0}}$ (3) $\displaystyle
2:\mathbf{d}_{k}$ $\displaystyle\rightarrow\mathbf{d}_{k+1},$
$\displaystyle\phantom{0}\textbf{via: }R_{i+1}$ $\displaystyle
3:\mathbf{d}_{k+1}$ $\displaystyle\rightarrow\mathbf{d}_{k+2},$
$\displaystyle\phantom{0}\textbf{via: }R_{i+2}$ $\displaystyle\cdots$
$\displaystyle n:\mathbf{d}_{k+n}$ $\displaystyle\rightarrow\mathbf{p}_{g},$
$\displaystyle\phantom{0}\textbf{via: }R_{g\phantom{+0}}$

Therefore, akin to informed geometric planners (e.g., informed RRT* [3]),
herein the semantic planner adds an additional layer of information that the
geometric planner can profit from to find a solution faster. The subproblems
are solved by the Subproblem Solver in conjunction with the geometric planner.
The resulting individual path segments are joined to a final, global path. If
the resulting path requires updating, e.g., due to a blocked path, the
Subproblem Solver can efficiently reevaluate the changed or newly created
subproblems.

###  3.3 Global Map

Instead of relying on raw sensor readings, S-Nav features a global map
reconstruction module that builds an accurate global map from S-Graphs data,
which in turn is generated on the fly by the robot, resp. provided to the
robot if the environment is already fully mapped. The global map is kept
relatively simple, i.e., not featuring obstacles as this problem is more
effectively handled by the reactive planner on a local map.

S-Graphs provides the planes associated with each room, including the doorways
that mark the connection between two rooms. In the first step, for each room,
the vertical planes (walls) are converted to a closed 2D contour which
encompasses the free space within a room. Room contours serve two purposes:
First, to restrict the geometric planner’s sampler to sample only in areas
that effectively contribute to the final path. Second, to build an optimistic,
clutter-free (yet accurate), signed distance field representation of the
physical environment that forms the basis for the geometric planner.

Doorways of a certain width and located at a certain point are added between
the two closest walls of the two associated rooms. Just like contours, they
are also part of the signed distance field generation process.

###  3.4 Geometric Search

a) IRRT*b) IRRT*+S-Graphsc) IRRT*+S-Graphs+SPSRoomsGeometric pathS-Graphs
path+contoursDoorwaysSamplesGoalStart

Figure 3: a) The naïve approach using IRRT* creates an excessive amount of
samples and yet ends up with a suboptimal path. b) Restricting IRRT* to sample
within the rooms that are part of the optimal path greatly enhances the
solution with fewer samples. c) Restricting IRRT* to sample within the rooms
parts of the solution and decomposing the overall problem in subproblems
yields the best result (least amount of samples) as it effectively exploits
the rapid convergence of IRRT* for the resulting simpler problems.

The geometric search within S-Nav features state-of-the-art geometric planners
provided by the Open Motion Planning Library (OMPL). Within S-Nav we use
sampling-based planners (e.g., PRM, RRT, IRRT*), that create random (sometimes
with a heuristic) samples within the valid bounds of the state space. A
priori, for any given problem, the whole global map has to be considered.
Therefore, a large number of samples is required to find the optimal path.
Constraining the sampler to sample within the rooms that have to be visited
along the semantic path, greatly enhances the convergence rate of the planner
as no samples are wastefully created in areas that are of no interest.
Furthermore, using the SPS that decomposes the global problem into a set of
local problems effectively exploits the rapid convergence of certain planning
algorithms (e.g., IRRT*) for the resulting simpler type of problem. The
problem is illustrated in fig. 3.

##  4 Evaluation

###  4.1 Methodology

A synthetic map
($$17\text{\,}\mathrm{m}$\crossproduct$15\text{\,}\mathrm{m}$$) with $8$ rooms
and $10$ doorways was created and passed to the recently presented iS-Graphs
[6], an S-Graphs extension that supports architectural (BIM) data. Within this
environment, the three cases (IRRT*, IRRT*+S-Graphs, and IRRT*+S-Graphs+SPS)
shown in fig. 3 were benchmarked by performing $1000$ queries for each. As
OMPL termination criteria, a timeout of $0.1\text{\,}\mathrm{s}$ was
specified. The timeout is equally divided over all subproblems for the test
series involving the SPS. Recorded were the number of samples created within
the allocated time as well as the final path length. The measurements were
performed on a workstation equipped with an Intel Core i9-11950H.

###  4.2 Results and Discussion

The results are given in fig. 4. It is clear that IRRT* alone delivers the
most inconsistent results with the widest spread. On average, it also had the
least number of samples generated. Restricting the sampled regions with the
S-Graphs knowledge, significantly improved the consistency of the results.
Further, using S-Graphs and the SPS in combination with IRRT* yielded
consistently the shortest path, and was also able to generate significantly
more samples. The higher number of samples is caused by the comparatively
cheaper state and motion validity checks based on the contours rather than the
signed distance field alone.

Figure 4: Results: The semantic planner greatly improves the consistency of
the results by reliably finding the shorter path compared to plain IRRT*. The
solution involving IRRT*, S-Graphs, and the SPS was able to create
significantly more samples in the allocated time.

##  5 Conclusion

Leveraging the geometric-semantic knowledge contained in S-Graphs for planning
can greatly enhance the performance of the underlying geometric planner.
Herein, we presented S-Nav, a novel semantic-geometric planner that features a
hierarchical planner architecture that showed to significantly improve
planning speed, resp., the consistency of the generated paths within a given
timeframe. Furthermore, we showed that decomposing the global problem into a
set of local problems can be used to effectively leverage the rapid
convergence of (informed) sampling-based planners.

## References

  * Bavle et al. [2022a] Hriday Bavle, Jose Luis Sanchez-Lopez, Muhammad Shaheer, Javier Civera, and Holger Voos.  Situational Graphs for Robot Navigation in Structured Indoor Environments.  _IEEE Robotics and Automation Letters_ , 7(4):9107–9114, oct 2022a.  ISSN 2377-3766.  doi:10.1109/LRA.2022.3189785.  URL https://ieeexplore.ieee.org/document/9826367/.
  * Bavle et al. [2022b] Hriday Bavle, Jose Luis Sanchez-Lopez, Muhammad Shaheer, Javier Civera, and Holger Voos.  S-Graphs+: Real-time Localization and Mapping leveraging Hierarchical Representations.  (1):1–10, dec 2022b.  URL http://arxiv.org/abs/2212.11770.
  * Gammell et al. [2014] Jonathan D. Gammell, Siddhartha S. Srinivasa, and Timothy D. Barfoot.  Informed RRT*: Optimal sampling-based path planning focused via direct sampling of an admissible ellipsoidal heuristic.  In _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 2997–3004. IEEE, sep 2014.  ISBN 978-1-4799-6934-0.  doi:10.1109/IROS.2014.6942976.  URL https://ieeexplore.ieee.org/document/6942976/.
  * Oleynikova et al. [2017] Helen Oleynikova, Zachary Taylor, Marius Fehr, Roland Siegwart, and Juan Nieto.  Voxblox: Incremental 3D Euclidean Signed Distance Fields for on-board MAV planning.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , volume 2017-Septe, pages 1366–1373. IEEE, sep 2017.  ISBN 978-1-5386-2682-5.  doi:10.1109/IROS.2017.8202315.  URL http://ieeexplore.ieee.org/document/8202315/.
  * Oleynikova et al. [2018] Helen Oleynikova, Zachary Taylor, Roland Siegwart, and Juan Nieto.  Sparse 3d topological graphs for micro-aerial vehicle planning, 2018.
  * Shaheer et al. [2023] Muhammad Shaheer, Jose Andres Millan-Romera, Hriday Bavle, Jose Luis Sanchez-Lopez, Javier Civera, and Holger Voos.  Graph-based Global Robot Localization Informing Situational Graphs with Architectural Graphs.  mar 2023.  URL http://arxiv.org/abs/2303.02076.
  * Zhang and Singh [2017] Ji Zhang and Sanjiv Singh.  Low-drift and real-time lidar odometry and mapping.  _Autonomous Robots_ , 41(2):401–416, feb 2017\.  ISSN 0929-5593.  doi:10.1007/s10514-016-9548-2.  URL http://link.springer.com/10.1007/s10514-016-9548-2.


# Kernels of splitting homomorphisms

Michael R. Klug

###### Abstract

Lei and Wu have given a description of the second homotopy group of a closed
orientable 3-manifold in terms of the kernels of the epimorphisms from the
fundamental group of a Heegaard splitting surface onto the fundamental groups
of the two handlebody sides. In this note, we give a geometric derivation of
this result and collect some observations about the relation between the
various groups and the topology of the 3-manifold and the Heegaard splitting.

##  1 Introduction

The role of the fundamental group in 3-dimensional topology is central. In
[17], Stallings put forth an approach to the 3-dimensional Poincaré conjecture
by way of the group theory associated to a Heegaard splitting of a homotopy
3-sphere. This approach was furthered by Jaco [6] who proved that several
group-theoretic statements involving free groups and surface groups were
equivalent to the Poincaré conjecture.

Let $M$ be a compact orientable 3-manifold and let $M=H_{\alpha}\cup
H_{\beta}$ be a Heegaard splitting of $M$ with $\Sigma=H_{\alpha}\cap
H_{\beta}$ an orientable surface of genus $g$. Fix a point $\ast$ in $\Sigma$
and consider the associated pushout diagram of groups, with morphisms induced
by inclusions.

${\pi_{1}(\Sigma,\ast)}$${\pi_{1}(H_{\alpha},\ast)}$${\pi_{1}(H_{\beta},\ast)}$${\pi_{1}(M,\ast)}$

The maps from the surface group to the free groups of the handlebodies are
surjective as any curve in a handlebody can be made to miss the spine of the
handlebody and thereby can be homotoped to the boundary. The map
$\phi:\pi_{1}(\Sigma)\to\pi_{1}(H_{\alpha})\times\pi_{1}(H_{\beta})$ is the
_splitting homomophism_ associated to the Heeagard surface $\Sigma$. Jaco
showed how all of the topology of $M$ is encoded in the splitting homomorphism
[6]. From here out, we will not generally mention the basepoints explicitly.

Building on the approach of Stallings and Jaco, Hempel [5] showed the
condition in the following theorem is equivalent to the Poincaré conjecture
and therefore, by the work of Perelman [14], [15], it follows:

######  Theorem 1.

(Perelman)

For all integers $g\geq 0$ and any pair of surjective homomorphisms
$\phi_{1},\phi_{2}:\pi_{1}(\Sigma)\to F_{g}\times F_{g}$ where $\Sigma$ is a
genus $g$ closed orientable surface, and $F_{g}$ is a rank $g$ free group,
there is an isomorphism

${\pi_{1}(\Sigma)}$${\pi_{1}(\Sigma)}$$\scriptstyle{\cong}$

and a pair of isomorphisms

${F_{g}}$${F_{g}}$$\scriptstyle{\cong}$

such that the following commutes

${\pi_{1}(\Sigma)}$${F_{g}\times F_{g}}$${\pi_{1}(\Sigma)}$${F_{g}\times
F_{g}}$$\scriptstyle{\phi_{1}}$$\scriptstyle{\cong}$$\scriptstyle{\cong}$$\scriptstyle{\cong}$$\scriptstyle{\phi_{2}}$

In other words, there is a unique surjective homomorphism $\pi_{1}(\Sigma)\to
F_{g}\times F_{g}$ up to pre- and post-composition with automorphisms (where
post-composition is by automorphisms that are a product of automorphisms on
the $F_{g}$ factors).

More recently, a similar group-theoretic statement equivalent to the smooth
4-dimensional Poincaré conjecture has been given by Abrams, Gay, and Kirby
[1]. In considering these connections, we wanted to understand how the basic
topological invariants of a space can be seen from the perspective of
splitting homomorphisms. Let
$K_{\alpha}=\ker(\pi_{1}(\Sigma)\to\pi_{1}(H_{\alpha}))$ and
$K_{\beta}=\ker(\pi_{1}(\Sigma)\to\pi_{1}(H_{\beta}))$. If the handlebodies
$H_{\alpha}$ and $H_{\beta}$ are described using a Heegaard diagram, then
$K_{\alpha}$ and $K_{\beta}$ are normally generated by the curves describing
$H_{\alpha}$ and $H_{\beta}$, respectively. In this note, we give a geometric
proof of a result of Lei and Wu [10] that shows how to compute $\pi_{2}(M)$ in
terms of $K_{\alpha}$ and $K_{\beta}$ (see Theorem 3).

##  2 Preliminary lemmas and general remarks

We say a surface $F$ is of finite type if $\pi_{1}(F)$ is finitely generated,
otherwise we say that $F$ has infinite type. Surfaces of finite type are up to
homeomorphism determined by their genus, number of boundary components, and
number of punctures. Surfaces of infinite type also admit a classification in
terms of their genus, number of boundary components, and space of ends [9],
[16].

For a surface of finite type $F$ with genus greater than or equal to 2,
together with a choice of hyperbolic metric on $F$, there are at most finitely
many many closed geodesics of length less than a given constant
$L\in\mathbb{R}$ [3].

######  Theorem 2.

If $g\geq 2$, then $K_{\alpha}\cap K_{\beta}$ is a not-finitely-generated free
group.

###### Proof.

In [7], it is shown that $K_{\alpha}\cap K_{\beta}\neq 1$. Let
$\widetilde{\Sigma}$ be the cover of $\Sigma$ corresponding to $K_{\alpha}\cap
K_{\beta}$. Then $\widetilde{\Sigma}$ is noncompact since $K_{\alpha}\cap
K_{\beta}$ has infinite index in $\pi_{1}(\Sigma)$. Therefore, $K_{\alpha}\cap
K_{\beta}$ is free since noncompact surfaces have free fundamental groups [8].
Since $g\geq 2$, the surface $\widetilde{\Sigma}$ obtains a hyperbolic metric
by pulling back a hyperbolic metric on $\Sigma$. Since $K_{\alpha}\cap
K_{\beta}$ is normal, the deck translations of the covering act on
$\widetilde{\Sigma}$ as isometries. Let $\gamma$ be a closed geodesic in
$\widetilde{\Sigma}$. Then all of the infinite translates of $\gamma$ have the
same length as $\gamma$. Thus, $\widetilde{\Sigma}$ is of infinite type, by
the preceding discussion, and therefore $K_{\alpha}\cap K_{\beta}$ is not
finitely generated. ∎

Recall that a 3-manifold $M$ is called reducible if there is an embedded
sphere in $M$ that does not bound a 3-ball in $M$, and irreducible otherwise.
Equivalently, by the Sphere Theorem [13], $M$ is irreducible if and only if
$\pi_{2}(M)=0$. A Heegaard splitting $M=H_{\alpha}\cup_{\Sigma}H_{\beta}$ is
called reducible if there is an essential simple closed curve in $\Sigma$ that
bounds embedded disks in both $H_{\alpha}$ and $H_{\beta}$; and irreducible
otherwise. Haken’s lemma [4] asserts that any Heegaard splitting of a
reducible 3-manifold is reducible.

######  Proposition 1.

The Heegaard splitting $M=H_{\alpha}\cup_{\Sigma}H_{\beta}$ is reducible if
and only if the subgroup $K_{\alpha}\cap K_{\beta}$ contains a nontrivial
element that can be represented by an embedded curve. If $M$ is reducible,
then $K_{\alpha}\cap K_{\beta}$ contains a nontrivial element that can be
represented by an embedded curve.

###### Proof.

The first follows immediately from the definitions. The second part follows
from Haken’s lemma. ∎

Note that given any irreducible Heegaard splitting of an 3-manifold, the group
$K_{\alpha}\cap K_{\beta}$ is a nontrivial subgroup of a surface group that,
by Proposition 1, cannot contain any elements that can be represented by
embedded curves. Other examples of this phenomenon, in fact finite index
examples, are known (see [12], [11]).

The following lemma is immediate upon considering the “$4g$-gon with a hole”
picture of a genus $g$ surface with one boundary component.

######  Lemma 1.

Let $X$ be a topological space with basepoint $p\in X$ and let
$A,B\leq\pi_{1}(X,p)$ be two normal subgroups. An element
$\gamma\in\pi_{1}(X,p)$ is in $[A,B]$ if an only if there is a continuous map
$f\colon(S,\partial S)\to(X,\gamma)$ where $S$ is a genus $g$ orientable
surface with one boundary component such that the images of the curves $a_{i}$
and $b_{j}$ in Figure 1 are in the normal subgroups $A$ and $B$ respectively
for $1\leq i,j\leq g$.

Figure 1: This figure shows a genus 2 example. The basepoint is assumed to be
on the boundary of the surface. The curves in red are the $a_{i}$ curves and
the curves in blue are the $b_{j}$ curves. Note here that $A$ and $B$ are
assumed to be normal subgroups, so the lack of a basepoint for the curves is
irrelevant – i.e., if some choice of arcs to the basepoint results in based
curves that are in $A$ and $B$, respectively, then so too does any other
choice of arcs.

Let $F$ be a surface in a 3-manifold $M$. By adding a tube to $F$, we mean
creating a new surface $F^{\prime}$, that is obtained from $F$ by taking the
symmetric difference of $F$ with the boundary of an embedded
$(D^{2}\times[0,1],D^{2}\times\\{0,1\\})\to(M,F)$ such that the image of
$D^{2}\times(0,1)$ is disjoint from $F$.

######  Lemma 2.

Let $F$ be a compact surface in $S^{3}$. There is a surface $F^{\prime}$
obtained from adding tubes to $F$ such that $F^{\prime}$ is isotopic to the
standard genus $g$ Heegaard splitting surface of $S^{3}$. Moreover, if $F$ is
a surface in $B^{3}$ with $F\cap\partial B^{3}$ connected, then we can add
tubes to $F$ to obtain a standard surface as in Figure 2.

###### Proof.

Recall that any 3-manifold $M$ with nonempty boundary has a handle
decomposition with a 0-handle, 1-handles, and then 2-handles – one way of
seeing this is that $M$ admits a Morse function that is constant on the
boundary and that has increasing index critical points. By carving out the
2-handles, we obtain a handlebody and therefore, from every 3-manifold with
boundary, we can obtain a handlebody if we carve out enough tubes. Now let $F$
be a surface in $S^{3}$. Note that carving tubes out of one side of $F$
corresponds to adding 1-handles to the other side, and notice that adding
1-handles to a handlebody produces another handlebody (with higher genus).
Therefore, by adding tubes to one side of $F$, we can obtain a new surface
that bounds a handlebody on one side, and by adding tubes to the other side,
we can obtain a surface that bounds handlebodies on both sides. The moreover
statement is obtained from this result by adding a ball with a standard
equator disk to the $B^{3}$ containing $F$, with the boundary of the disk
glued to the boundary of $F$, and performing the above argument with all of
the tubes not intersecting this trivial disk-ball pair. ∎

Figure 2: This figure shows a genus 3 example of the trivial surface in a
3-ball. The boundary of the surface is the equator and the surface is the
result of stabilizing (as with Heegaard splittings) the equitorial disk three
times.

######  Lemma 3.

Let $\Sigma$ be a Heegaard splitting surface of a closed orientable 3-manifold
$M$, $F$ be a compact connected surface, and $f\colon F\to M$ be a continuous
map. Then $f$ can be homotoped so that $f^{-1}(\Sigma)$ is connected.

###### Proof.

First, homotope $f$ such that it is an immersion transverse to $\Sigma$. Let
$\gamma_{1}$ and $\gamma_{2}$ be two distinct connected components of
$f^{-1}(\Sigma)$ such that there is an arc $a\subset F$ with endpoints in
$\gamma_{1}$ and $\gamma_{2}$ respectively and such that $f(a)$ is entirely
contained in $H_{\alpha}$ or $H_{\beta}$. Note that any properly embedded arc
in a handlebody can be homotoped relative to its endpoints so as to be
contained in the boundary of the handlebody. Therefore, by homotoping $f$
close to $a$ following this homotopy of $a$ into $\Sigma$ we obtain a new map
with one fewer connected component in the inverse image of $\Sigma$. Repeating
this process then gives the result. ∎

##  3 $\pi_{2}(M)$ from splitting homomorphisms

The following result is given by Lei and Wu in [10], where it is stated that
it follows from general methods in [2]. Here we provide a hands-on geometric
argument.

######  Theorem 3.

(Lei and Wu) For every closed $3$-manifold $M$, there is an isomorphism of
abelian groups

$\pi_{2}(M)\cong(K_{\alpha}\cap K_{\beta})/[K_{\alpha},K_{\beta}]$

Let $p\colon\pi_{1}(\Sigma)\twoheadrightarrow\pi_{1}(M)$ be the surjection
induced by inclusion. Then $(K_{\alpha}\cap K_{\beta})/[K_{\alpha},K_{\beta}]$
has a well-defined $\pi_{1}(M)$-action given by
$g\cdot[\gamma]=[g^{\prime}\gamma g^{\prime-1}]$ for $g\in\pi_{1}(M)$ with
$g^{\prime}$ a choice of preimage $p(g^{\prime})=g$ and
$[\gamma]\in(K_{\alpha}\cap K_{\beta})/[K_{\alpha},K_{\beta}]$. With respect
to this action, the above isomorphism is a $\mathbb{Z}\pi_{1}(M)$-module
isomorphism.

###### Proof.

We first define a map $\phi\colon K_{\alpha}\cap K_{\beta}\to\pi_{2}(M)$.
Notice that, given a curve $\gamma\in K_{\alpha}\cap K_{\beta}$ then by the
definition of $K_{\alpha}$ and $K_{\beta}$, there exist disks
$D_{\alpha}\subset H_{\alpha}$ and $D_{\beta}\subset H_{\beta}$ such that
$\gamma=\partial D_{\alpha}=\partial D_{\beta}$. If $D^{\prime}_{\alpha}$ and
$D^{\prime}_{\beta}$ are other such disks, then the spheres
$D_{\alpha}\cup_{\gamma}D_{\beta}$ and
$D^{\prime}_{\alpha}\cup_{\gamma}D^{\prime}_{\beta}$ are homotopic with the
homotopy fixing the basepoint, since $\pi_{2}$ of a handlebody is trivial and
so the union of the disks can be extended to a map from a 3-ball. Therefore,
we have a map $\phi$ which is seen to be a group homomorphism.

The surjectivity of $\phi$ is the content of Lemma 3 in the case where $F$ is
a sphere.

We will now show that the map $\phi$ descends to the quotient $K_{\alpha}\cap
K_{\beta}/[K_{\alpha},K_{\beta}]$. As a preliminary remark, observe that the
commutator $[K_{\alpha},K_{\beta}]$ is a subgroup of the intersection
$K_{\alpha}\cap K_{\beta}$, because the kernels $K_{\alpha},K_{\beta}$ are
normal subgroups of $\pi_{1}(\Sigma)$. To see that
$[K_{\alpha},K_{\beta}]\leq\ker(\phi)$ note that by Lemma 1, there exists a
surface $S$ as in Figure 1 and a continuous map $f\colon(S,\partial
S)\to(\Sigma,\gamma)$ such that $f(a_{i})\in K_{\alpha}$ and $f(b_{j})\in
K_{\beta}$. Therefore, we have immersed disks $D^{\alpha}_{i},D^{\beta}_{i}$
with $\partial D^{\alpha}_{i}=f(a_{i})$ and $\partial D^{\beta}_{j}=f(b_{j})$.
We thus have a capped surface (i.e., the result of adding to the $a_{i}$ and
$b_{j}$ curves in Figure 1) mapping into $M$ with the boundary mapping to
$\gamma$ – call the image $\hat{S}$. Now consider a neighborhood of $\hat{S}$
which is topologically a ball. Thus, we obtain a map of a 2-sphere whose
equator maps to $\gamma$ and such that the two disks bounding the equator map
to disks $D_{\alpha}$ and $D_{\beta}$ in $H_{\alpha}$ and $H_{\beta}$,
respectively. Then $\phi(\gamma)=0$ since $D_{\alpha}\cup_{\gamma}D_{\beta}$
bounds a ball.

We now prove injectivity of the resulting map

$\phi:(K_{\alpha}\cap K_{\beta})/[K_{\alpha},K_{\beta}]\to\pi_{2}(M)$

Suppose that $\gamma\in\ker(\phi)$ and let $g\colon B^{3}\to M$ with
$g(\partial B^{3})=D_{\alpha}\cup_{\gamma}D_{\beta}=\phi(\gamma)$. If
$g^{-1}(\Sigma)$ is standard as in Figure 2, then, by Lemma 1, we have that
$\gamma\in[K_{\alpha},K_{\beta}]$. If $g^{-1}(\Sigma)$ is not standard, then,
by homotoping $g$, we can add tubes to $g^{-1}(\Sigma)$, since such a tube
maps to a thickened arc in $M$ that is either contained in $H_{\alpha}$ or
$H_{\beta}$ (where as in the proof of Lemma 2, every arc can be homotoped to
lie in the boundary). Therefore, by Lemma 2, we can make $g^{-1}(\Sigma)$
standard. Thus $\ker(\phi)\leq[K_{\alpha},K_{\beta}]$ and therefore $\phi$
induces an isomorphism of abelian groups $K_{\alpha}\cap
K_{\beta}/[K_{\alpha},K_{\beta}]\cong\pi_{2}(M)$.

Now, consider the action of $\pi_{1}(M)$ on $(K_{\alpha}\cap
K_{\beta})/[K_{\alpha},K_{\beta}]$ as in the statement of the result. By our
construction of $\phi$ and the definition of the action of $\pi_{1}(M)$ on
$\pi_{2}(M)$, we see that $\phi$ is in fact an isomorphism of
$\mathbb{Z}[\pi_{1}(M)]$-modules, thus proving the result. ∎

######  Remark 1.

We have not been discussing the subgroup
$K_{\alpha}K_{\beta}\leq\pi_{1}(\Sigma)$, which is another subgroup of
interest (here $K_{\alpha}K_{\beta}$ denoted the join of $K_{\alpha}$ and
$K_{\beta}$ – i.e., the smallest subgroup containing boht of them). Note that
$\pi_{1}(\Sigma)/K_{\alpha}K_{\beta}\cong\pi_{1}(M)$ (see [17]) and the cover
of $\Sigma$ corresponding to $K_{\alpha}K_{\beta}$ is the preimage of $\Sigma$
in the universal cover for $M$. It then follows, as in Theorem 2, that either

  1. 1.

$\pi_{1}(M)$ is finite and $K_{\alpha}K_{\beta}$ is isomorphic to the
fundamental group of a closed orientable surface, or

  2. 2.

$\pi_{1}(M)$ is infinite and $K_{\alpha}K_{\beta}$ is an infinitely generated
free group.

## References

  * [1] Aaron Abrams, David T. Gay, and Robion Kirby.  Group trisections and smooth 4-manifolds.  Geom. Topol., 22(3):1537–1545, 2018.
  * [2] Ronald Brown and Jean-Louis Loday.  Van Kampen theorems for diagrams of spaces (with an appendix by M. Zisman).  Topology, 26(3):311–335, 1987.
  * [3] Peter Buser.  Geometry and spectra of compact Riemann surfaces.  Springer Science & Business Media, 2010.
  * [4] Wolfgang Haken.  Some results on surfaces in 3-manifolds.  Studies in modern topology, 5:39–98, 1968.
  * [5] John Hempel.  3-manifolds, Reprint of the 1976 original.  AMS Chelsea Publishing, Providence, RI, 2004.
  * [6] William Jaco.  Heegaard splittings and splitting homomorphisms.  Trans. Amer. Math. Soc., 144:365–379, 1969.
  * [7] William Jaco.  On certain subgroups of the fundamental group of a closed surface.  Proc. Cambridge Philos. Soc., 67:17–18, 1970.
  * [8] Ingebrigt Johansson.  Topologische Untersuchungen über unverzweigte Überlagerungsflächen.  Number 1. Kommisjon hos J. Dybwad, 1932.
  * [9] Béla Kerékjártó.  Vorlesungen über Topologie, volume 8.  J. Springer, 1923.
  * [10] Fengchun Lei and Jie Wu.  The intersecting kernels of Heegaard splittings.  Algebr. Geom. Topol., 11(2):887–908, 2011.
  * [11] Charles Livingston.  Maps of surface groups to finite groups with no simple loops in the kernel.  J. Knot Theory Ramifications, 9(8):1029–1036, 2000.
  * [12] Justin Malestein and Andrew Putman.  On the self-intersections of curves deep in the lower central series of a surface group.  Geom. Dedicata, 149:73–84, 2010.
  * [13] Christos D Papakyriakopoulos.  On Dehn’s lemma and the asphericity of knots.  Proceedings of the National Academy of Sciences of the United States of America, 43(1):169, 1957.
  * [14] Grisha Perelman.  The entropy formula for the Ricci flow and its geometric applications.  arXiv preprint math/0211159, 2002.
  * [15] Grisha Perelman.  Ricci flow with surgery on three-manifolds.  arXiv preprint math/0303109, 2003.
  * [16] Ian Richards.  On the classification of noncompact surfaces.  Transactions of the American Mathematical Society, 106(2):259–269, 1963.
  * [17] John R. Stallings.  How not to prove the Poincaré conjecture.  Ann. of Math. Studies, pages 83–88, 1966.