# DMON: A Simple yet Effective Approach for Argument Structure Learning

###### Abstract

Argument structure learning (ASL) entails predicting relations between
arguments. Because it can structure a document to facilitate its
understanding, it has been widely applied in many fields (medical, commercial,
and scientific domains). Despite its broad utilization, ASL remains a
challenging task because it involves examining the complex relationships
between the sentences in a potentially unstructured discourse. To resolve this
problem, we have developed a simple yet effective approach called Dual-tower
Multi-scale cOnvolution neural Network (DMON) for the ASL task. Specifically,
we organize arguments into a relationship matrix that together with the
argument embeddings forms a relationship tensor and design a mechanism to
capture relations with contextual arguments. Experimental results on three
different-domain argument mining datasets demonstrate that our framework
outperforms state-of-the-art models. The code is available at
https://github.com/VRCMF/DMON.git.

Keywords: argument structure learning, argument mining

DMON: A Simple yet Effective Approach for Argument Structure Learning

Wei Sun, Mingxiao Li, Jingyuan Sun, Jesse Davis, Marie-Francine Moens
---
Department of Computer Science, KU Leuven
Celestijnenlaan 200A 3001 Heverlee, Belgium
Sun.Wei, Mingxiao.Li, Jingyuan.Sun<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Argument structure learning (ASL) Moens (2013); Lawrence and Reed (2020)
involves detecting and tagging relationships between argumentative components
in a text. Figure 1 shows an illustrative example of an argumentative
structure for a medical report where pairs of sentences are annotated with
whether there is a supportive or attacking relationship between them. This
problem is a cornerstone in the semantic analysis of natural language text
because it helps to elucidate the relational structure. Consequently, it helps
facilitate more accurate and deeper comprehension of text and hence plays a
critical role in various NLP applications such as patient-generated content
analysis Mayer et al. (2020); Stylianou and Vlahavas (2021), legal reasoning
Poudyal et al. (2020), and opinion mining Niculae et al. (2017).

Despite of its broad application, solving the ASL task is still challenging
due to the complexity of text structures and diversity of relationships.
Moreover, real-world data often contains inconsistencies and are largely
unstructured. Fully understanding the relationship between two arguments often
require contextual knowledge from other arguments, or even their
relationships.

A key challenge posed by ASL is that fully understanding the relationship
between two arguments often requires capturing contextual knowledge about
other arguments and their relationships. In Figure 1, to classify the
relationship $\textbf{C}\rightarrow\textbf{D}$, examples of contextual
argument relationships are $\textbf{A}\rightarrow\textbf{D}$ and
$\textbf{B}\rightarrow\textbf{D}$. Mayer et al. (2020), Stylianou and Vlahavas
(2021) and Galassi et al. (2021) tried to conduct pairwise relation
classification for ASL without contextual information, yielding sub-optimal
classification performance. A more recent attempt by Hua and Wang (2022)
encodes the contextual arguments with a transformer architecture. This helped
to improve its accuracy, but they still ignored the relationships between
contextual arguments.

Figure 1: A simple example of an argumentative structure showing attack
(orange arrow) and support (green arrow) relationships.

In this paper, we for the first time propose to exploit contextual argument
relationships to solve the ASL task. As shown in Figure 2, we represent the
argument structure as a relationship tensor to capture the contextual
information about argument relationships. This also allows us to naturally
model the relationships between pairs of arguments that can be bidirectional
and asymmetric. We propose a bidirectional learning approach that uses a
separate model for each direction. Moreover, training is also hampered by the
fact that there is limited labeled data for ASL problems due to the high
annotation costs. Therefore, we propose a cropping strategy that randomly
samples a subtensor that maintains the ordering of the selected relationships.

Figure 2: We select argument $C$ as the observation object. This example shows
the correlation between the head (red) and tail (blue) relationship
information and a relationship tensor. Each element in this relationship
tensor is the concatenation of two arguments.

The main contributions of this paper are the following:

  * •

We propose a novel approach called DMON to encode contextual arguments and
their relationships by connecting the argument structure with a relationship
tensor.

  * •

We propose a bidirectional learning mechanism that allows distinguishing head
and tail arguments in a relationship.

  * •

We design a cropping strategy to handle the scarcity of training data.

  * •

Experimental results on three different-domain argument mining datasets show
that our method outperforms state-of-the-art models for the ASL task.

First, we discuss related works in section 2. Next, in section 3, we provide a
detailed description of the DMON framework. Then, in section 4, we present
experimental results on two argument mining datasets from different domains
and conduct an ablation study to analyze the proposed framework and its
outcomes. Finally, in section 5, we summarize this paper.

Figure 3: The structure of Multi-scale Residual Convolution Neural Network
(DMON) during training. During testing, no cropping mechanism is used. In this
example, the yellow and green cells in a prediction matrix correspond to
attack and support relations.)

## 2\. Related Work

### 2.1. Argument Structure Learning

Argument Structure Learning is a challenging but essential task in text mining
Moens (2013); Lawrence and Reed (2020). Most papers tackle argument
relationship classification by performing pairwise ASL and classifying two
concatenated sentences due to it simplicity and effectiveness Mayer et al.
(2020); Stylianou and Vlahavas (2021); Galassi et al. (2021).

In Hua and Wang (2022), argument components and their contextual sentences
were encoded together using a RoBERTa encoder to obtain neighboring arguments
information. The experimental results demonstrate that incorporating
contextual information enhances performance. This method primarily relies on
self-attention to capture relationships between the concatenated sentences.
However, self-attention can be computationally expensive as it involves
computing a pairwise similarity matrix for every token in all argument
components. In this paper we explicitly model contextual argumentative
relationships and, given that a convolution operation can effectively capture
input-data features Alzubaidi et al. (2021); Andreoli (2019); Dumoulin and
Visin (2016), we utilize convolutional modules to represent a pair of
arguments and contextual pairs of arguments.

### 2.2. Structured Learning in NLP

Structured learning (SL) also called structured prediction plays a crucial
role in many NLP tasks. It models complex relationships and dependencies
within text data to improve the performance of various discourse-related
applications, such as sentiment analysis Ein-Dor et al. (2022), and
summarization Xu et al. (2020). In argument mining, Niculae et al. (2017) use
a structured support vector machine, while Bao et al. (2021) implement
transition-based dependency parsing to reveal the argumentative structure.

## 3\. Method

Problem Setting: Given a document containing $n$ sentences, we treat each one
as a potential argument and use $A$ to denote the argument set. We convert $A$
into the set of ordered pairs $C_{A}=\\{(a_{i},a_{j})|a_{j}\in
A~{}\text{and}~{}a_{i}\in A\\}\in\mathbb{R}^{n^{2}}$. The goal of ASL is to
classify all relations in the set $C_{A}$ with the domain-specific labels
contained in the given dataset.

Method Overview: We introduce a Dual-tower Multi-scale cOnvolution neural
Network (DMON) for the ASL task. The model has four components (Figure 3).
Firstly, we use an encoder to extract pairwise argument representations. Given
that an argumentative graph is a directed acyclic graph (DAG), potential
argumentative relationships can be represented as an asymmetric relationship
matrix, or as a relationship tensor when it includes the pairwise argument
embeddings. Secondly, during training, a cropping strategy selects sub-tensors
from the relationship tensor. Thirdly, a bidirectional learning mechanism is
applied to the cropped relationship tensors to capture contextual arguments
and their relationships. Finally, we employ label fusion to merge two
predicted label matrices into one label matrix. During testing, we feed full
relationship tensors instead of cropped tensors into the model.

### 3.1. Pairwise Arguments Representation

Following the literature Mayer et al. (2020); Hua and Wang (2022); Stylianou
and Vlahavas (2021), we treat each sentence as a potential argument. To
capture pairwise interactions, we create all possible pairs $(a_{i},a_{j})$
and concatenate them with a special token placed between them. Next, we use a
linkBERT model to encode the paired arguments into an average pooled embedding
with dimensionality $d$. We organize these into a tensor
$\mathbf{H}\in\mathbb{R}^{n\times n\times d}$. Figure 2 illustrates how to
transform the sentences of a discourse into a relationship tensor. Each cell
in the relationship tensor represents the concatenation of a coupled argument,
which typically consists of two elements: the first element is known as the
head argument, while the second element is referred to as the tail argument.

### 3.2. Cropping Strategy

We use a cropping strategy to mitigate scarcity of labeled training data.
During each training iteration, we sample a new sub-tensor
$\mathbf{H}^{\prime}\in\mathbb{R}^{m\times m\times d}$ from $\mathbf{H}$.
Concretely, we sample $m$ indices $\\{i_{0},i_{1},\ldots,i_{m-1}\\}$ without
replacement from $\\{0,1,\ldots,n-1\\}$ where $m<n$ and $m$ is called the
window size. Now we describe in mathematical notation how these indices induce
a sub-tensor. $\mathbf{H}^{{}^{\prime}}_{j,k}$ represents an element of sub-
tensor $\mathbf{H}^{\prime}$ whose row and column indexes are $j$ and $k$, and
$\mathbf{H}^{\prime}_{j,k}=\mathbf{H}_{i_{j},i_{k}}$. The cropping strategy
keeps the discourse order of arguments and relations of the full relationship
tensor in its rows and columns. Because the cropping strategy looks at the
sub-graph induced by the selected vertices, it maintains the alignment of
contextual arguments (graph’s vertices) and relations (graph’s edges). The
cropped relationship tensors are resampled in each training iteration, which
can be viewed as a form of data augmentation.

### 3.3. Bidirectional Learning Mechanism

We have developed a bidirectional learning mechanism to predict the labels of
a relationship between its head and tail. Head and tail relationship
information is captured by applying a multi-scale (1D) convolution on the
relationship tensor both horizontally and vertically, respectively.

Because the relationship tensor can take into account both short- and long-
distance relationships between the argument sentences in a discourse, we
leverage a multi-scale residual module (MSRM). Prior work Li and Yu (2020)
reveals that a multi-filter convolutional layer (similar to an Inception
block) can capture varied relationships. However, Li et al. (2018) stated that
the Inception architecture leads to the underutilization of local features.
Therefore, we choose the MSRM Li et al. (2018) as our base module and use
different kernel sizes for the convolutional filters to capture the short- and
long distance relationships. Because of the asymmetry of the relationships, we
apply the MSRM both horizontally and vertically on the relationship tensor to
generate predictions while taking into account the contextual argument
structure of the discourse. Figure 4 illustrates the structure of the MSRM.

Figure 4: The structure of the multi-scale residual module (MSRM). k1, k2, k3
and k4 represent the kernel size of the respective 1D convolution layer.

We illustrate here how to capture the contextual arguments and head
relationship information using the horizontal branch during trainng with a
cropped tensor. Firstly, we obtain the representations of the first row of the
relationship tensor,
$\mathbf{h}_{1}=\mathbf{H}^{{}^{\prime}}[1,:,:]\in\mathbb{R}^{1\times m\times
d}$ and pass them into the MSRM. We denote the 1D convolutional layer with
kernel size $k_{x}$ as $\mathcal{F}_{k_{x}}(\cdot)$. We omit the bias item in
the equations and the output $\mathbf{o}_{1}\in\mathbb{R}^{1\times m\times d}$
is:

$\displaystyle\mathbf{S}_{1}$
$\displaystyle=\operatorname{ReLU}(\mathcal{F}_{k_{1}}(\mathbf{h_{1}})),$ (1)
$\displaystyle\mathbf{P}_{1}$
$\displaystyle=\operatorname{ReLU}(\mathcal{F}_{k_{2}}(\mathbf{h_{1}})),$ (2)
$\displaystyle\mathbf{S}_{2}$
$\displaystyle=\operatorname{ReLU}(\mathcal{F}_{k_{2}}(\operatorname{Concat}(\mathbf{S}_{1},\mathbf{P}_{1}))),$
(3) $\displaystyle\mathbf{P}_{2}$
$\displaystyle=\operatorname{ReLU}(\mathcal{F}_{k_{3}}(\operatorname{Concat}(\mathbf{S}_{1},\mathbf{P}_{1}))),$
(4) $\displaystyle\mathbf{o}_{1}$
$\displaystyle=\mathcal{F}_{k_{4}}(\operatorname{Concat}(\mathbf{S}_{2},\mathbf{P}_{2}))+\mathbf{h_{1}},$
(5)

where $\operatorname{ReLU}(\cdot)$ represents the ReLU activation function and
$\operatorname{Concat}(\cdot)$ is the feature concatenation operation. We
repeat the above calculations for $m$ times and we concatenate all output
features to get the resulting tensor of the head convolution
$\mathbf{O}^{m}_{h}=\operatorname{Concat}(o_{1},\cdots,o_{m})\in\mathbb{R}^{m\times
m\times d}$. Thirdly, a linear classifier layer consisting of flattened logits
are applied to produce the predictions. On the horizontal branch, this is
$\mathbf{\hat{y}}^{h}\in\mathbb{R}^{l}$ where $l$ represents the label space
dimension. The vertical branch uses the same operations as above but the
logits are denoted as $\mathbf{\hat{y}}^{t}\in\mathbb{R}^{l}$.111Note that
during testing (inference stage) in the above computations $m$ is replaced by
$n$ (number of paired arguments of the full text) and
$\mathbf{H}^{{}^{\prime}}$ by $\mathbf{H}$.

The cross-entropy losses of the horizontal and vertical branches are
calculated as follows:

$\displaystyle\mathcal{L}_{h}=-\frac{1}{m^{2}}\sum_{i=1}^{m^{2}}\sum_{j=1}^{l}{y}_{i,j}\log(\hat{y}^{h}_{i,j}),$
(6)
$\displaystyle\mathcal{L}_{td}=-\frac{1}{m^{2}}\sum_{i=1}^{m^{2}}\sum_{j=1}^{l}{y}_{i,j}\log(\hat{y}^{t}_{i,j}),$
(7)

where $\mathbf{y}\in\mathbb{R}^{l}$ are one-hot encoded vectors representing
the ground-truth labels. We adopt joint training for the two branch losses and
the joint training loss is defined as:

$\displaystyle\mathcal{L}=\lambda_{h}\mathcal{L}_{h}+\lambda_{t}\mathcal{L}_{t},$
(8)

where $\lambda_{h}$ and $\lambda_{t}$ are scaling factors for the horizontal
and vertical branches, respectively.

### 3.4. Label Fusion

During testing, a confidence voter fuses the label predictions from the
horizontal and vertical branches. Inspired by Vyas et al. (2018) and Weng et
al. (2023), we measure the confidence scores of logits by the difference
between top-$1$ and top-$2$ probabilities. Assume that we have logits
$\widetilde{\mathbf{y}^{h}}\in\mathbb{R}^{l}$ and
$\widetilde{\mathbf{y}^{t}}\in\mathbb{R}^{l}$, the difference between the
top-$1$ and top-$2$ probabilities for $\widetilde{\mathbf{y}^{h}}$ is
$\widetilde{\mathbf{c}^{h}}$:

$\displaystyle\widetilde{\mathbf{s}^{h}}$
$\displaystyle=\operatorname{Softmax}(\widetilde{\mathbf{y}^{h}}),$ (9)
$\displaystyle\widetilde{\mathbf{c}^{h}}$
$\displaystyle=\operatorname{Topk}(\widetilde{\mathbf{s}^{h}},k_{1})-\operatorname{Topk}(\widetilde{\mathbf{s}^{h}},k_{2}),$
(10)

where $\operatorname{Topk}(\cdot,k)$ returns the $k-$th largest elements from
the given input and $\operatorname{Softmax}(\cdot)$ is the softmax
function.222Results with other normalization functions (e.g., L1, L2, min-max)
did not improve results. Secondly, we get the confidence score for the
vertical branch $\widetilde{\mathbf{c}^{t}}$ by following the above
computations. Thridly, we applied the argmax operation to get the predictions
of two branches, $\overline{\mathbf{y}^{h}}\in\mathbb{R}^{n\times n}$ and
$\overline{\mathbf{y}^{t}}\in\mathbb{R}^{n\times n}$. We merge two matrices
into $\overline{\mathbf{y}}\in\mathbb{R}^{n\times n}$ based on confidence
scores $\widetilde{\mathbf{c}^{h}}$ and $\widetilde{\mathbf{c}^{t}}$. The
merging process for the $\overline{{y}}_{i,j}$ is:

$\displaystyle\overline{{y}}_{i,j}=\begin{cases}\overline{{y}^{h}}_{i,j},&\text{if
}\widetilde{{s}^{h}}_{i,j}\geq\widetilde{{s}^{t}}_{i,j}\\\
\overline{{y}^{t}}_{i,j},&\text{otherwise}\end{cases}$ (11)

Models | Neoplasm | Glaucoma | Mixed
---|---|---|---
F1 | S-F1 | A-F1 | U-F1 | F1 | S-F1 | A-F1 | U-F1 | F1 | S-F1 | A-F1 | U-F1
AMPERE++ | 63.73 | - | - | - | - | - | - | - | - | - | - | -
Roberta | 67.00 | - | - | - | 66.00 | - | - | - | 69.00 | - | - | -
AMCT-Sci | 68.16 | 59.99 | 49.12 | 95.45 | 62.28 | 64.71 | 24.78 | 95.24 | 69.43 | 55.31 | 58.00 | 94.76
TransforMED | 69.96 | 58.72 | 55.65 | 95.51 | 69.72 | 65.32 | 47.00 | 96.88 | 71.82 | 57.14 | 63.41 | 94.90
RESATTARG | 70.92 | 52.77 | 65.38 | 94.54 | 68.40 | 54.73 | 56.00 | 94.36 | 67.66 | 49.62 | 59.09 | 94.21
DMON |  76.30 $\pm$0.71 |  68.25 $\pm$0.98 |  64.13 $\pm$0.47 |  97.33 $\pm$0.08 |  74.16 $\pm$1.10 |  73.16 $\pm$1.17 |  53.41 $\pm$0.63 |  97.27 $\pm$0.06 |  74.07 $\pm$1.11 |  68.35 $\pm$1.42 |  54.05 $\pm$0.74 |  97.08 $\pm$0.09
Table 1: Experimental results of argument structure learning in terms of
macro-F1 scores on three AbstRCT medical datasets. S-F1, A-F1, U-F1 and F1
refer to the average macro-F1 score of the support relation, of the attack
relation, of no-relation, and of their average, respectively. For DMON the
mean results over 5 runs with variance are shown.

## 4\. Experiments

The goal is to evaluate the Macro F1 score (it returns objective results on
imbalanced ASL datasets) of the detection and correct classification of
argumentative relationships in a discourse, to compare the results with the
results of state-of-the-art baselines, and to assess the influence of the
proposed components of the DMON model.

### 4.1. Datasets

We conduct experiments in the medical, legal and scientific domains,
represented by the AbstRCT Mayer et al. (2020), the Cornell eRulemaking Corpus
(CDCP) Niculae et al. (2017), and the SciDTB Yang and Li (2018) datasets.

AbstRCT: The AbstRCT corpus consists 659 medical documents about the treatment
for specific disease (neoplasm, glaucoma, hypertension, hepatitis-b,
diabetes). Following Mayer et al. (2020), the corpus is divided into three
datasets based on disease category: neoplasm, glaucoma, and mixed. The
neoplasm (neo) dataset contains 350 documents for training, 50 for validation
and 100 notes for testing. The neoplasm train set is used as train set for the
glaucoma (gla) and mixed (mix) dataset, which each have 100 instances for
testing. The labels of the relationships of the three AbstRCT datasets are
‘attack’, ‘support’, and ‘un-related’.

SciDTB: The SciDTB (SCI) dataset includes 1049 scientific abstracts collected
from the ACL Anthology. It consists of 743 examples for training, 154 samples
for validation and 152 for testing. The dataset has more fine-grained
discourse relationship categories while the number of labels is 27.

CDCP: The CDCP dataset contains 731 user comments about consumer debt
collection practices from an eRulemaking website and include 581 examples for
training and 150 for testing. Labels of the CDCP dataset are ‘related’ and
‘un-related’.

### 4.2. Experimental Set-up

Model Settings: The maximum sequence length is $128$. The combination of
kernel sizes for the multi-scale convolution module is set to
$\\{k_{1},k_{2},k_{3},k_{4},k_{5}\\}=\\{7,5,5,3,1\\}$. The window size for the
cropping is $13$. $\lambda_{h}$ and $\lambda_{t}$ are set to 0.5. $k_{1}$ and
$k_{2}$ is set as 1 and 2 in the label fusion part.

Training Details: We fine-tune the BioLinkBERT Yasunaga et al. (2022) for the
AbstRCT dataset and LinkBERT Yasunaga et al. (2022) for the SciDTB and CDCP
dataset. For the training, all baseline models and our framework are trained
with FP16. We train all models on a NVIDIA GeForce RTX 3090 GPU. All models
use the AdamW optimizer Loshchilov and Hutter (2019) with a learning scheduler
initialized at $2e^{-5}$ and linearly decreased to 0.

Models | Full-F1 | F1 | R-F1 | U-F1
---|---|---|---|---
RESARG | - | 67.60 | 38.99 | 96.20
RESATTARG | - | 73.64 | 50.00 | 97.28
BERT | 32.83 | 73.89 | 50.47 | 97.30
AMCT-Sci | 34.58 | 75.89 | 54.61 | 97.17
Roberta | 35.96 | 75.88 | 54.25 | 97.51
DMON |  48.37 $\pm$0.54 |  87.36 $\pm$0.25 |  77.03 $\pm$0.36 |  98.70 $\pm$0.02
Table 2: Experimental results of argument structure learning in terms of
macro-F1 scores on the SciDTB datasets. For DMON the mean results over 5 runs
with variance are shown. Full-F1, R-F1, U-F1 and F1 refer to the average
macro-F1 score of the full label space, the related relation, of no-relation,
and of related and no-relation, respectively. For DMON the mean results over 5
runs with variance are shown.

### 4.3. Baselines

We consider models that classify the argument relationships given a
representation of the pairs of sentences obtained with a pretrained encoder.

AMPERE++ Hua and Wang (2022) uses a Roberta model to concatenate $20$
neighbouring sentences with an argument and only predicts whether this
argument is a head or tail argument. The authors did not name this model so we
call it AMPERE++.

BERT Devlin et al. (2019) fine-tunes a pretrained BERT model to encode an
argument pair and predict its relationship.

Roberta Liu et al. (2019) fine-tunes a pretrained Roberta model to represent a
pair of sentences and classifies the relationships.

AMCT-Sci Stylianou and Vlahavas (2021) is similar to Roberta, but encodes
argument pairs with a domain-specific BERT
model.333https://huggingface.co/allenai/scibert_scivocab_uncased

TransforMED Stylianou and Vlahavas (2021) integrates a medical knowledge
system to extract medical entities from arguments. The authors inject medical
knowledge into their model by concatenating features of arguments and medical
entities.

We also consider models that use attention mechanisms to model relationships
between arguments.

RESARG Galassi et al. (2021) used a BiLSTM to extract textual feature and then
applied a residual neural network to deal with the ASL task.

RESATTARG Galassi et al. (2021) extended RESARG with a coarse-grained parallel
co-attention mechanism to predict argumentative relations.

TSP-PLBA Morio et al. (2020) consists of task-specific parameterization (TSP)
and proposition-level biaffine attention (PLBA) to capture argument structure
from documents. TSP encodes the arguments while PLBA predicts argument
relations by using a biaffine scoring function.

We then consider models that train a transition-based dependency parser.

BERT-Trans Bao et al. (2021) leverage the BERT language model to obtain
representation and propose a neural transition-based model to generate a
sequence of actions (shift, delete-delay, delete, right-arc, right-arc-delay,
and left-arc) that build an argument structure (predicted nodes and
relations).

Models | F1 | R-F1 | U-F1
---|---|---|---
TSP-PLBA | - | 34.00 | -
AMPERE++ | 63.10 | - | -
RESATTARG | 64.40 | 30.60 | 98.30
BERT-Trans | 67.80 | 37.30 | 98.30
DMON |  68.14 $\pm$0.45 |  38.26 $\pm$0.74 |  98.37 $\pm$0.06
Table 3: Experimental results of argument structure learning in terms of
macro-F1 scores on the CDCP datasets. For DMON the mean results over 5 runs
with variance are shown. R-F1, U-F1 and F1 refer to the average macro-F1 score
of the related relation, of no-relation, and of their average, respectively.

### 4.4. Results and Discussion

We run each model five times with different seeds and report the average
macro-F1 scores and their variance obtained on the three absRCT datasets the
CDCP dataset and the SciDTB dataset.

AbsRCT: Table 1 shows that the DMON outperforms all baselines on all average
F1 scores. Compared with the state-of-the-art model TransforMED, our model
improves the average macro-F1 scores by $6.34$, $5.76$, and $2.25$ percentage
points on the Neoplasm, Glaucoma, and Mixed datasets, respectively. Even
though TransforMED explicitly injects external medical knowledge, our approach
still performs better.

SciDTB: Table 2 shows that the DMON outperforms baseline models by a large
margin when evaluated on the SciDTB dataset. Compared to the Roberta model,
our model improves the average macro-F1 scores by $12.41$, $11.48$, and
$22.78$ percentage points on Full-F1, F1, and R-F1 scores.

CDCP: Table 3 shows that DMON also achieves the best performance in terms of
average macro-F1 score on the CDCP dataset. Compared to the BERT-Trans model
our method improves the averaged macro-F1 and R-F1 by $0.79$ and $1.7$
percentage points, respectively. Compared with the BERT-Trans, our model is
simple (can be applied to other pairwise classification models) and can
achieve better performance. Additionally, BERT is the transformer encoder but
the proposed neural transition-based model is to generate a sequence of
actions (shift, delete-delay, delete, right-arc, right-arc-delay, and left-
arc) that build an argument structure (predicted nodes and relations). To
train this transition system, they need to convert argument structure learning
data into the transition-based structure data. This preprocessing adds
complexity.

Discussion: We observe that the performance gains of DMON oompared to state-
of-the-art baselines are different when analyzing the results obtained on the
three domains. For instance, when comparing with baseline RESATTARG, DMON
improves the macro-F1 scores by 13.92, 5.38, 3.74 percentage points on SciDTB,
Neoplasm, and CDCP datasets, respectively. The AbstRCT (of which Neoplasm is a
part) and CDCP are imbalanced and have a high ratio of sentence pairs that
exhibit no argumentative relationship ("unrelated" relationship). This could
be a reason why DMON has somewhat lower performance gains. On the other hand,
AbstRCT and CDCP have few relationship types and it might be that the
baselines already deal with these in a satisfactory way, while they have more
difficulties with the 27 relationship types of SciDTB. For this more difficult
case of argumentative structure learning, DMON has the highest gains in
performamce and improves the average macro-F1 scores by $12.41$, $11.48$, and
$22.78$ percentage points on all 27 relationship type, on the "unrelated"
class and on the other 26 relationships, respectively.

Large Language Models: As large language models capture the attention of the
NLP communities, we conduct a comparison of LLMs with the proposed algorithm.
We pick the GPT 3.5 turbo (gpt-3.5-turbo-0613), a widely used LLM, and
validate it on the ASL task. We use the in-context prompt learning (ICL) Brown
et al. (2020) to validate the ASL task. We found the zero-shot ICL is much
worse than the few-shot ICL, so we reported the results of the GPT 3.5 turbo
by using the few-shot ICL method. Experimental results show the number of the
demonstration samples is 2. We report the Macro-F1 scores of the GPT 3.5 on
the AbstRCT (gla), AbstRCT (neo), AbstRCT (mix), CDCP and SciDTB datasets,
which are 12.03, 13.84, 11.19, 11.89, and 17.24, respectively. Results of the
GPT 3.5 turbo are much lower than the fine-tuned models, i.e., our DMON model,
whose scores are 73.16, 76.30, 74.07, 87.36, and 68.14. We think the reason
why the GPT 3.5 performs badly on the ASL datasets is because LLMs fail to
deal with tasks which require complex reasoning ability.

(a)

(b)

(c)

(d)

(e)

Figure 5: Macro-F1 scores when changing window size of contextual arguments.

### 4.5. Ablation Study and Analysis of Results

We conduct ablation experiments to study DMON’s components and analyze them.

#### 4.5.1. Bidirectional Learning Mechanism and Confidence Voter

Table 4 shows results when ablating the bidirectional learning mechanism and
confidence voter. The F1 scores obtained on all datasets are largely reduced
if we remove the bidirectional learning mechanism and confidence voter. The
macro-F1 scores on all three datasets (Neoplasm, Glaucoma, and CDCP) decrease
when we just use the predictions of one of the two branches (i.e., head or
tail relationships) and use no confidence voter. The same pattern holds when
we directly remove one of two branches when training the model. The macro-F1
score is reduced on the Mixed dataset when we remove the confidence voter and
use tail prediction. The macro-F1 shows the same pattern when we only remove
the head branch.

Model | Neo | Gla | Mix | CDCP | SCI
---|---|---|---|---|---
DMON | 76.30 | 74.16 | 74.07 | 68.14 | 87.36
w/o Voter (h) | 74.31 | 72.04 | 73.42 | 67.50 | 87.08
w/o Voter (t) | 75.74 | 73.83 | 74.84 | 67.50 | 86.58
w/o T | 73.74 | 70.10 | 71.40 | 65.39 | 85.75
w/o H | 75.62 | 71.48 | 74.88 | 66.55 | 87.08
w/o H+T | 69.10 | 71.28 | 70.35 | 56.24 | 74.27
Table 4: Experimental results of argument structure learning in terms of
average macro-F1 scores on the Neoplasm (Neo), Glaucoma (Gla), Mixed (Mix),
and CDCP datasets. w/o Voter (h) or Voter (t) removes the confidence voter and
leverages head or tail prediction, respectively. w/o T or H removes either the
head branch or tail branch when training the model. w/o H+T completely removes
the bidirectional learning mechanism.

#### 4.5.2. Cropping Strategy

We analyze several aspects of the cropping strategy.

Training with Different Cropped Tensors: Figure 5 compares results of DMON
using cropped tensors with the results of using the full relationship tensor
as input. We observe that DMON using cropped tensors outperforms DMON with the
full tensor on the four datasets. Using the cropping strategy improves the
macro-F1 score of our framework by $0.2$, $0.95$, $4.25$, $1.38$, and $0.18$
percentage points on Neoplasm, Glaucoma, Mixed and SciDTB datasets,
respectively. The cropped tensors offer more variation in the training data,
and its representations contribute to the generalization capabilities of the
model.

Model | Neo | Gla | Mix | CDCP | SCI
---|---|---|---|---|---
$\textbf{DMON}_{\text{win=13}}$ | 76.30 | 74.16 | 74.07 | 68.14 | 87.36
$\textbf{DMON}_{full}$ | 76.10 | 73.21 | 69.82 | 66.76 | 87.18
Table 5: Results of the cropping strategy with window size of 13
($\textbf{DMON}_{\text{win=13}}$) and of using the complete relationship
tensor ($\textbf{DMON}_{full}$) during training.

Information Alignment: To study the information alignment of the cropping with
the original relationship tensor, we develop two shuffle approaches called
order shuffle (ord) and random shuffle (rad). ord scatters the order of the
arguments, that is, horizontal and vertical indexes of the cropped
relationship tensor are shuffled. rad randomly chooses pairwise samples to
fill the cropped relationship tensor. Figure 6 shows an example illustrating
the ord and rad approaches.

Figure 6: An example that illustrates the order shuffle and random shuffle.
For simplicity we only show the relationship matrix.

Table 6 shows that the macro-F1 score decreases by using the rad method.
Therefore, it is important to keep head and tail relationships aligned, that
is, correctly representing the asymmetric relationships between arguments.
Table 6 also reveals that the order of the argumentative sentence pairs in the
relationship tensor is important as it implicitly captures the coherence of
the discourse.

original | ord | rad | Neo | Gla | Mix | CDCP | SCI
---|---|---|---|---|---|---|---
✓ | ✗ | ✗ | 76.30 | 74.16 | 74.07 | 68.14 | 87.36
✗ | ✓ | ✗ | 75.20 | 68.68 | 72.44 | 61.07 | 75.88
✗ | ✗ | ✓ | 71.43 | 74.31 | 72.85 | 65.86 | 86.69
✗ | ✓ | ✓ | 59.98 | 55.08 | 60.02 | 50.78 | 66.27
Table 6: An experiment for exploring the information alignment of the cropping
strategy. original represents DMON without any shuffling strategies.

Contextual Windows Size: The results demonstrate that encoding contextual
arguments and their relationships is beneficial. Figure 5 shows the macro-F1
scores by changing the window size of the cropped relationship tensor for the
evaluated datasets. The macro-F1 score curve shows an upward trend by encoding
more neighboring arguments and relationships, but slightly decreases when
considering the full discourse (Table 5).

## 5\. Conclusion

In this paper, we have proposed a novel framework called the Dual-tower Multi-
scale cOnvolution neural Network (DMON) to deal with the ASL task that in a
flexible way can learn the argumentative DAG structure taking into account
contextual argumentative relationships. A sentence or clause on its own seldom
fulfils an argumentative role in a discourse, it is only when paired with
another sentence and in the context of other sentences that its argumentative
role becomes apparent. In an argumentative DAG structure a sentence can have
multiple parents and children, and our model can deal with this flexibility.
We conduct experiments on four datasets covering the medical, legal and
scientific domains, namely abstRCT, CDCP SciDTB and achieve new state-of-the-
art performance, when compared to several strong baselines. Furthermore, we
perform ablations and in-depth analyses to prove the effectiveness of each
component of our model.

## Limitations

The limitation of our paper are reflected as follows: 1) Computation
limitations prevented us from fully exploring the effectiveness of the self-
attention mechanism in argument structure learning. In future work, we might
leverage more powerful GPUs when implementing a multi-head self-attention
mechanism and thoroughly investigate its impact. Nevertheless the performance
of our model is better than previous state-of-the-art models that use
attention and are computationally more expensive. 2) We will test whether our
bidirectional learning mechanism can be embedded to other pairwise
classification models in NLP to improve their performance. In the future, we
will leverage balancing approaches, such as focal loss and resampling, to
alleviate this problem.

## 6\. Acknowledgements

This research was funded by the CHIST-ERA projects ANTIDOTE (ERA-NET CHIST-ERA
IV FET PROACT JTC 2019) and iTRUST (ERA-NET CHIST-ERA JTC 2021).

## 7\. Bibliographical References

##

  * Alzubaidi et al. (2021) Laith Alzubaidi, Jinglan Zhang, Amjad J Humaidi, Ayad Al-Dujaili, Ye Duan, Omran Al-Shamma, José Santamaría, Mohammed A Fadhel, Muthana Al-Amidie, and Laith Farhan. 2021.  Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions.  _Journal of big Data_ , 8:1–74.
  * Andreoli (2019) Jean-Marc Andreoli. 2019.  Convolution, attention and structure embedding.  _arXiv preprint arXiv:1905.01289_.
  * Bao et al. (2021) Jianzhu Bao, Chuang Fan, Jipeng Wu, Yixue Dang, Jiachen Du, and Ruifeng Xu. 2021.  A neural transition-based model for argumentation mining.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6354–6364.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ding et al. (2022) Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. 2022.  Scaling up your kernels to 31x31: Revisiting large kernel design in cnns.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11963–11975.
  * Dumoulin and Visin (2016) Vincent Dumoulin and Francesco Visin. 2016.  A guide to convolution arithmetic for deep learning.  _arXiv preprint arXiv:1603.07285_.
  * Eger et al. (2017) Steffen Eger, Johannes Daxenberger, and Iryna Gurevych. 2017.  Neural end-to-end learning for computational argumentation mining.  _arXiv preprint arXiv:1704.06104_.
  * Ein-Dor et al. (2022) Liat Ein-Dor, Ilya Shnayderman, Artem Spector, Lena Dankin, Ranit Aharonov, and Noam Slonim. 2022.  Fortunately, discourse markers can enhance language models for sentiment analysis.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 10608–10617.
  * Galassi et al. (2018) Andrea Galassi, Marco Lippi, and Paolo Torroni. 2018.  Argumentative link prediction using residual networks and multi-objective learning.  In _Proceedings of the 5th Workshop on Argument Mining_ , pages 1–10.
  * Galassi et al. (2020) Andrea Galassi, Marco Lippi, and Paolo Torroni. 2020.  Attention in natural language processing.  _IEEE transactions on neural networks and learning systems_ , 32(10):4291–4308.
  * Galassi et al. (2021) Andrea Galassi, Marco Lippi, and Paolo Torroni. 2021.  Multi-task attentive residual networks for argument mining.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Hua and Wang (2022) Xinyu Hua and Lu Wang. 2022.  Efficient argument structure extraction with transfer learning and active learning.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 423–437, Dublin, Ireland. Association for Computational Linguistics.
  * Imrich et al. (2008) Wilfried Imrich, Sandi Klavzar, and Douglas F Rall. 2008.  _Topics in graph theory: Graphs and their Cartesian product_.  CRC Press.
  * Jiang et al. (2023) Yuchen Eleanor Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Mrinmaya Sachan, and Ryan Cotterell. 2023.  Discourse centric evaluation of machine translation with a densely annotated parallel corpus.  _arXiv preprint arXiv:2305.11142_.
  * Lawrence and Reed (2020) John Lawrence and Chris Reed. 2020.  Argument mining: A survey.  _Computational Linguistics_ , 45(4):765–818.
  * Li and Yu (2020) Fei Li and Hong Yu. 2020.  Icd coding from clinical text using multi-filter residual convolutional neural network.  In _proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 8180–8187.
  * Li et al. (2018) Juncheng Li, Faming Fang, Kangfu Mei, and Guixu Zhang. 2018.  Multi-scale residual network for image super-resolution.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 517–532.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Luo et al. (2016) Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. 2016.  Understanding the effective receptive field in deep convolutional neural networks.  _Advances in neural information processing systems_ , 29.
  * Mayer et al. (2020) Tobias Mayer, Elena Cabrio, and Serena Villata. 2020.  Transformer-based argument mining for healthcare applications.  In _ECAI 2020_ , pages 2108–2115. IOS Press.
  * Mochales and Moens (2011) Raquel Mochales and Marie-Francine Moens. 2011.  Argumentation mining.  _Artificial Intelligence and Law_ , 19:1–22.
  * Moens (2013) Marie-Francine Moens. 2013.  Argumentation mining: Where are we now, where do we want to be and how do we get there? In _Proceedings of the 5th 2013 Forum on Information Retrieval Evaluation, FIRE ’13, New Delhi, India, December 4-6, 2013_ , pages 2:1–2:6. ACM.
  * Morio et al. (2020) Gaku Morio, Hiroaki Ozaki, Terufumi Morishita, Yuta Koreeda, and Kohsuke Yanai. 2020.  Towards better non-tree argument mining: Proposition-level biaffine parsing with task-specific parameterization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3259–3266.
  * Niculae et al. (2017) Vlad Niculae, Joonsuk Park, and Claire Cardie. 2017.  Argument mining with structured SVMs and RNNs.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 985–995, Vancouver, Canada. Association for Computational Linguistics.
  * Palau and Moens (2009) Raquel Mochales Palau and Marie-Francine Moens. 2009.  Argumentation mining: the detection, classification and structure of arguments in text.  In _Proceedings of the 12th international conference on artificial intelligence and law_ , pages 98–107.
  * Poudyal et al. (2020) Prakash Poudyal, Jaromír Šavelka, Aagje Ieven, Marie Francine Moens, Teresa Goncalves, and Paulo Quaresma. 2020.  Echr: legal corpus for argument mining.  In _Proceedings of the 7th Workshop on Argument Mining_ , pages 67–75.
  * Stylianou and Vlahavas (2021) Nikolaos Stylianou and Ioannis Vlahavas. 2021.  Transformed: End-to-end transformers for evidence-based medicine and argument mining in medical literature.  _Journal of Biomedical Informatics_ , 117:103767.
  * Vyas et al. (2018) Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L Willke. 2018.  Out-of-distribution detection using an ensemble of self supervised leave-out classifiers.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 550–564.
  * Weng et al. (2023) Juanjuan Weng, Zhiming Luo, Dazhen Lin, Shaozi Li, and Zhun Zhong. 2023.  Boosting adversarial transferability via fusing logits of top-1 decomposed feature.  _arXiv preprint arXiv:2305.01361_.
  * Wolf and Gibson (2006) Florian Wolf and Edward Gibson. 2006.  _Coherence in natural language: data structures and applications_.  MIT Press.
  * Xiong et al. (2019) Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang. 2019.  Modeling coherence for discourse neural machine translation.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 7338–7345.
  * Xu et al. (2020) Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Discourse-aware neural extractive text summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5021–5031, Online. Association for Computational Linguistics.
  * Yang and Li (2018) An Yang and Sujian Li. 2018.  SciDTB: Discourse dependency TreeBank for scientific abstracts.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 444–449, Melbourne, Australia. Association for Computational Linguistics.
  * Yasunaga et al. (2022) Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022.  LinkBERT: Pretraining language models with document links.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8003–8016, Dublin, Ireland. Association for Computational Linguistics.
  * Zhang et al. (2022) Haokui Zhang, Wenze Hu, and Xiaoyu Wang. 2022.  Parc-net: Position aware circular convolution with merits from convnets and transformer.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI_ , pages 613–630. Springer.


11institutetext: Photogrammetry and Remote Sensing, TUM School of Engineering
and Design, Technical University of Munich (TUM),
Munich, Germany - (thomas.froech, olaf.wysocki, ludwig.hoegner<EMAIL_ADDRESS>22institutetext: Department of Geoinformatics, University of Applied Science
(HM), Munich, Germany -<EMAIL_ADDRESS>

# Reconstructing façade details using MLS point clouds and Bag-of-Words
approach

Thomas Froech1  Olaf Wysocki1  Ludwig Hoegner1,2  Uwe Stilla1

###### Abstract

In the reconstruction of façade elements, the identification of specific
object types remains challenging and is often circumvented by rectangularity
assumptions or the use of bounding boxes. We propose a new approach for the
reconstruction of 3D façade details. We combine mobile laser scanning (MLS)
point clouds and a pre-defined 3D model library using a Bag of words (BoW)
concept, which we augment by incorporating semi-global features. We conduct
experiments on the models superimposed with random noise and on the TUM-FAÇADE
dataset [30]. Our method demonstrates promising results, improving the
conventional BoW approach. It holds the potential to be utilized for more
realistic facade reconstruction without rectangularity assumptions, which can
be used in applications such as testing automated driving functions or
estimating façade solar potential.

###### Keywords:

point clouds façade reconstruction Bag-of-Words Approach mobile laser scanning

##  1 Introduction

Figure 1: Overview of the approach, left side: training process, right side
inference(illustration based on [19])

Semantic 3D building models up to level of detail (LoD) 2 level are widely
used and available today [2]. LoD 3 models characterized by a higher level of
detail of their façade representation are scarce
111https://github.com/OloOcki/awesome-citygml. These are necessary for various
applications, such as testing automated driving functions or estimating façade
solar potential [29]. The primary challenges in developing 3D façade
reconstruction methods lie in the availability of street-level measurements
and complexity of detailed façade element reconstruction; which is often
circumvented by reconstructing the elements as bounding boxes [12]. Recently,
however, there has been an increase in the availability of high-accuracy,
street-level MLS point cloud data [30]. In addition, databases that contain
high-quality hand-modeled 3D façade details already exist [27]. These possess
the potential to bridge the data-gap and move beyond the assumption of
rectangularity and the use of bounding boxes.

In this paper, we propose an approach for the reconstruction of façade details
leveraging the accuracy of MLS point clouds and the ubiquity of high quality
3D façade elements’ libraries. Specifically, we employ an enhanced bag-of-
words (BoW) approach [7] to match measured façade elements with those from the
library, without the rectangular assumptions.

##  2 Related Work

###  2.1 LoD 3 Building Model Reconstruction

The reconstruction of LoD 3 building models has attracted attention over an
extended period of time [14, 28, 21, 23, 29]. Recent advancements have shown
that façade elements reconstruction can be robustly performed, and yet
identifying a specific object type remains challenging, for example,
distinguishing between rectangular and oval window types [29]. An example is
the study of Hoegner and Gleixner which aims at the extraction of a 3D model
of façades and windows from a MLS point cloud[13]. Their approach is based on
a voxel octree structure and visibility analysis. While they report a
detection rate of $86\%$, they simplify windows and façades by representing
them solely as rectangular shapes.

Other studies, such as that of Stilla and Tuttas, are devoted to the use of
airborne laser scanning (ALS) point clouds for the reconstruction of 3D
building models [28]. They introduce an approach for the generation of façade
planes with windows and the enrichment of a semantic city model with windows
from a multi-aspect oblique view ALS point cloud.

Following a different, image and deep learning-based approach for 3D model
reconstruction, Fan et al. propose VGI3D, an interactive platform for low-cost
3D building modeling from  volunteered geographic information (VGI) data using
convolutional neural network (CNN) s in 2021 [9]. Their easy to use,
lightweight, and quick application takes a small number images and a user’s
sketch of the façade boundary as an input [9]. For the automatic detection of
façade elements, the object detection CNN YOLO v3 [22] is utilized [9].

###  2.2 Bag of Words Approach

We use an adapted variant of the BoW concept in our study. The original BoW
approach is introduced by Salton and McGill in 1986 [25]. Csurka et al. apply
the BoW concept to images, where the detection and description of keypoints is
one of the foundations of their approach: bag-of-visual-words (BoVW) [7]. This
concept is also applied to point cloud data. Based on the Work of Xu et al. on
object classification of aerial images [31], Kang and Yang make use of a BoVW
approach to construct a supervoxel representation of raw point cloud data
[15]. The BoVW concept is further developed and adapted. Zhu et al. introduce
the local–global feature BoVW (LGFBoVW). They combine local and global
features on histogram level. Such a combination increases the robustness
compared to the traditional BoVW-approach [32].

##  3 Methodology

###  3.1 Overview

Figure 1 provides an overview of our proposed method. The training starts with
3D model preparation and sampling, where we create binary images from the
sampled point clouds. From these images, we extract and describe features,
which are then clustered to obtain a visual dictionary. By quantizing with the
Euclidean distance, we assign the closest codeword to every feature vector.
Next, we count each codewords occurrences, providing the representations of
the models as bags of codewords. During the inference, we represent the target
point clouds by the codebook. To compare both representations, we employ
histogram distances, where the model with the closest histogram distance to a
target point cloud is selected as the best match.

###  3.2 Sampling of the CAD models

A key element of the method we propose is the correspondence between features
extracted from the MLS point clouds and the CAD models. In order to improve
the comparability, we sample the CAD models with a sampling distance d to
obtain a point cloud. Figure 2 a) gives an example of such a point cloud. To
enable comparison to point clouds from MLS data, Figure 2 b) shows an
exemplary window that has been extracted from the TUM-Façade dataset.

The diversity of the CAD models does not allow for the same sampling distance
to be used for all models. The consequence of this could be sparse point
clouds or point clouds with too many points. Therefore, we choose d
individually for the CAD models to obtain a suitable point cloud. An
alternative approach could be to normalize the CAD models before the sampling.
gives an example of a point cloud sampled from a CAD model.

Figure 2: a) Point cloud sampled from CAD b) Point cloud extracted from the
TUM-Façade dataset

###  3.3 Binary Image Generation and Processing

We normalize the point clouds after outlier removal and downsampling. To
account for the decreasing point density with increasing height, we use an
outlier removal that depends on the average height of the windows from the MLS
point cloud.

Next, the point clouds are ortho-projected to a binary image ensuring its
frontal view. As illustrated in Figure 3, to enhance the extraction of
meaningful keypoints, we apply the standard image processing techniques.
Figure 4 a) shows that the majority of the identified keypoints are located at
semantically meaningful positions.

Figure 3: Exemplary image processing on a projected point cloud: a) projected
image of a point cloud, b) dilated image, c) edge detection (Laplace), d) line
simplification (Douglas-Peucker). Figure 4: Examples for feature extraction:
a) ORB-keypoints b) HOG-image

###  3.4 Feature Extraction

We use the Oriented FAST and Rotated BRIEF (ORB) [24] descriptor as a keypoint
point detector. This binary descriptor is based on the Binary Robust
Independent Features (BRIEF) descriptor [24, 4]. We choose this descriptor due
to its resistance to noise and higher computational speed compared to other
descriptors such as Scale Invariant Feature Transform (SIFT) [24, 17].

In our study, we use dense feature sampling as an alternative approach to the
interest point detection. This method is opposed to the concept of identifying
and describing key points. In dense feature sampling, descriptors are sampled
at points on a dense grid over the image, hence the name. We sample the ORB
descriptor for each of the points in the dense grid. This approach allows the
extraction of a large amount of information at the cost of higher
computational intensity [20].

We use the Histogram of Oriented Gradients (HOG) feature descriptor [8] to
incorporate semi-global information into the BoW approach. The fundamental
concept of this descriptor is the investigation of the gradients and their
orientation within the image on a dense grid [8]. Normalized histograms of
these gradients are established for each of these grid cells [8]. The
resulting one-dimensional vector characterizes the structure of the objects in
the image. [8, 18].

###  3.5 Incorporation of Semi-Global Features

Generally, shapes possess a limited number of features [3]. This poses
difficulties in extracting large numbers of distinct features. Semi-global
information can be used to mitigate the effects of this issue. However, semi-
global features, as well as global features, cannot be directly integrated
into the standard BoW method. Their (semi-) global uniqueness prevents the
establishment of a frequency of occurence. We propose a method to incorporate
HOG descriptors as semi-global information into the BoW-approach to overcome
this issue. Figure 4 b) gives an example of the information obtained with HOG.
A 2D diagram that displays the distribution of the gradients in the respective
cell is shown for every cell in the image, in which a gradient is present.
Figure 5 gives an overview of our concept. We establish a structure similar to
a histogram by considering each HOG descriptor variate as a separate histogram
bin, with the value of the bin equaling the value of the corresponding
variate. This structure comprises as many bins as HOG variates. We concatenate
the occurence histogram that we obtain from the BoW approach with the
histogram-similar structure constructed from the HOG-variates to a combined
histogram.

Figure 5: Incorporation of semi-global features

###  3.6 Clustering

For simplicity, we make use of the standard K-Means clustering algorithm in
our BoW approach. We treat every descriptor variate as an axis in this
clustering. Our clustering problem thus has the same dimensionality as the
extracted feature descriptor. With K-Means clustering, the number of clusters
n has to be determined in advance. The setting of this hyper-parameter is
critical for the performance of the BoW method since the meaningful assignment
of data points to cluster centers depends on it [15]. There appear only a few
empty or overcrowded cluster centers for this n. We acknowledge that there
might be better configurations for n, and that there are more sophisticated
clustering algorithms available that might lead to improved results.

###  3.7 Histogram Comparison

We use histogram distances to assess the similarity of the histograms within
our BoW approach. There is a large number of different histogram distances
available [5]. The specific choice of histogram distance employed holds
limited significance regarding the general approach that we introduce, thereby
allowing for arbitrary selection in our methodology. We make use of the
histogram distances that are introduced in the following

The Minkowski distance is given as [5]:

$D=\left(\sum_{i=1}^{n}\left|q_{i}-p_{i}\right|^{p}\right)^{\frac{1}{p}}$ (1)

The Minkowski distance is characterized by the parameter p. It can be
interpreted as a more general form of the Manhattan distance $(p=1)$, the
euclidean distance $(p=2)$ and the Chebyshev distance $(p=\infty)$ [5]. It can
be used to evaluate the similarity of two histograms by pairwise comparison of
the individual bins and subsequent accumulation of the obtained distances.

The Jensen-Shannon divergence is given as [1]:

$\text{{JSD}}(P,Q)=\frac{1}{2}\left(\text{{KL}}(P\|M)+\text{{KL}}(Q\|M)\right)$
(2)

where $\text{{KL}}(P\|Q)$ is the Kullback-Leibler-Divergence between the
probability distributions $P$ and $Q$. The Kullback-Leibler-Divergence given
as [1]:

$\text{{KL}}(P\|Q)=\sum_{i}P(i)\ln\left(\frac{P(i)}{Q(i)}\right)$ (3)

The Jensen-Shannon divergence, like the Kullback-Leibler divergence, is based
on the concept of entropy [5] and is used to quantify the similarity of two
probability distributions. The Jensen-Shannon divergence can be interpreted as
a symmetric version of the Kullback-Leibler divergence [5].

The Pearson Chi-Square-Distance between the probability distributions $P$ and
$Q$ is given as [5]:

$D\chi^{2}(P,Q)=\sum_{i}\frac{(P(i)-Q(i))^{2}}{P(i)}$ (4)

##  4 Experiments

We infered our method on the models, superimposed with random noise and on a
building from the labeled MLS point cloud of the TUM-FAÇADE data set [30]. To
ensure the comparability of our experiments, we consistently used the same
number of clusters n in all of them.

We found a suitable n in a heuristic way. We performed experimental clustering
and feature descriptor quantization for different values of n. We then
evaluated the frequency of occurrence of the respective cluster centers. We
find that in most of our cases 25 clusters give satisfactory results.

Figure 6: 3D window CAD models

We acquired four CAD models from the SketchUp 3D Warehouse library [27]
(Figure 6). We chose the models so that each window that is present in the
TUM-Façade dataset at least matches one of the models. We additionally chose
one window that is of octagon like shape as a model without a matching window
in the dataset.We manually edited some of the CAD models to add or remove
window bars.

Figure 7: Point clouds, sampled from CAD model: a) without removal of window
glass b) with removal of window glass

When sampling the CAD-models to a point cloud, transparency properties have to
be taken into account. As shown in Figure 7, we removed glass from all models
to exploit the presence or absence of window bars. In Figure 7 b) detailed
window bars become apparent due to glass components not being sampled. In
contrast, Figure 7 a) illustrates that when the glass components are sampled
using the same method as the rest of the window, it results in the window
appearing as an opaque object. With MLS scans, the laser beams usually
penetrate the glass and thus penetrate into the interior of the building. This
justifies the removal of glass from the CAD models.

We evaluate our results by determining the overall accuracy, the user’s
accuracy the producer’s accuracy, and Cohen’s kappa coefficient.

The overall accuracy (OA) is calculated by dividing the number of correctly
classified pixels by the total number of pixels [6]:

$\text{{OA}}=\frac{\text{{number of correctly classified
samples}}}{\text{{total sample number}}}$ (5)

The producer’s accuracy (PA) is used to determine the percentage of the ground
truth samples that is classified correctly [6]:

$\text{{PA}}=\frac{\text{{number of correctly classified samples of a class
X}}}{\text{{total number of ground truth samples in class X}}}$ (6)

The user’s accuracy (UA) is used to calculate the percentage of the
classification result of a class that is classified correctly [6]:

$\text{{UA}}=\frac{\text{{number of correctly classified samples of a class
X}}}{\text{{total number of samples classified as class X}}}$ (7)

Cohen’s kappa coefficient represents an alternative to using OA. It has a
value range from 0 to 1, where 0 represents complete randomness while 1 would
represent a perfect classifier [6]. It can be interpreted as a measure for the
concordance between the predicted class assignments and the class assignment
of the ground truth data [11]. It can be calculated from the confusion matrix
according to the following formula:

$\kappa=\frac{\text{{OA}}-\text{{RM}}}{1-\text{{RM}}}$ (8)

with the random match (RM):

$\text{{RM}}=\frac{\sum(\text{{product of row and column
sums}})}{(\text{{total sum}})^{2}}$ (9)

The implementation is available in the repository [10].

###  4.1 Experiments on models superimposed with random noise

We added random noise to the point clouds that we sample from the pre-
processed CAD models, as schematically described in Figure 8. We used the Chi-
Square histogram distance for these experiments. Results from these
experiments are summarized in Figure 9, Figure 10 and Table 1.

Figure 8: Schema: Addition of random noise to the point clouds sampled from
CAD Figure 9: Confusion matrices: a) ORB, b) ORB and HOG Figure 10: Confusion
matrix: ORB and HOG, noise level with factor 2

We find that incorporating HOG descriptors improves the matching quality.
Table 1 shows that the OA is improved from $0.47$ to $0.69$ this way. The
kappa-coefficient is improved from $0.36$ to $0.65$. We observe sensitivity
towards noise. By doubling the noise level, the kappa coefficient is
diminished to $0.43$, while the overall OA drops to $0.5$.

In addition, we identify a dependency of the matching quality on the type of
CAD model. Figures 9 and 10 show that the matching is more stable for the
arched and octagon shaped windows than for the rectangular and quadratic
windows.

Table 1: Results of the experiments with the models superimposed with random noise Experiment | Overall Accuracy | Kappa Coefficient
---|---|---
ORB, Chi-Square dist. | 0.47 | 0.36
ORB and HOG, Chi-Square dist. | 0.69 | 0.65
ORB and HOG, Chi-Square dist., noise $*2$ | 0.50 | 0.43

We conducted further experiments with different combinations of features and
histogram distances. More information on the experiments can be found here.
Figure 11 summarizes the standard deviations and the variances of the user’s
and producer’s accuracies for six of these experiments with different sets of
hyper parameters. We find that the arched window with no bars and the two
octagon-shaped windows are matched in the most stable manner compared to the
other window types.

Figure 11: Standard deviation and variance of user’s and producer’s accuracies
from experiments with six different combinations of hyper parameters.

###  4.2 Experiments on the TUM-FAÇADE dataset

Figure 12: a) ORB, Jensen-Shannon Divergence, b) ORB and HOG, Jensen-Shannon
Divergence, c) ORB and HOG, Minkowski-Distance

The tested façade only comprises rectangular and arched windows with window
bars (Figure 13). We observe an improvement of the matching quality with the
incorporation of HOG-descriptors, as the OA increases to $0.57$. Also, the
dependence on the histogram distance is shown in Table 2.

Table 2: Results of the experiments on the TUM-FAÇADE dataset [30] Experiment | Overall Accuracy
---|---
ORB, Jensen-Shannon Divergence | 0.36
ORB and HOG, Jensen-Shannon Divergence | 0.57
ORB and HOG, Minkowski-Distance | 0.41
Figure 13: Correctly (green) and falsely (red) matched windows Figure 14:
Results of the experiments on the TUM-FAÇADE dataset with dense feature
sampling

We observe a correlation of the matching quality with the building height in
Figure 13. This can presumably be attributed to the point density of the MLS
point cloud, which decreases with increasing altitude.

The experiment with ORB, HOG, and the Jensen-Shannon Divergence was also
conducted using dense feature sampling instead of extracting key points.
Figure 14 summarizes the results from this experiment. However, with an OA of
$0.45$, we observe a decrease in in accuracy.

##  5 Discussion

###  5.1 3D interest point detection

In our study, we employ 2D projections of point clouds to generate binary
images and extract different types of features from them. We adopt this
approach based on the intuition that the front view contains most of the
information that characterizes the window.

The point cloud data as well as the CAD models are three-dimensional.
Therefore, instead of a projection and subsequent extraction of 2D feature
descriptors, it seems obvious to use 3D interest point detection directly.
There is a number of such operators available. Examples are unsupervised
stable interest point detection (USIP) for points clouds [16] or Harris 3D for
meshes [26].

Assuming that the front view of a window contains the majority of information
that characterizes a specific window type, the potential benefit derived from
incorporating 3D information is expected to be marginal. The desired
correspondence between the point cloud that is sampled from the CAD model and
the MLS point cloud represents an obstacle when using such detectors. The
often irregular 3D shape of the windows from the MLS point clouds does not
necessarily correspond to the very regular 3D-shape of the windows that are
sampled from the CAD models. Therefore, the extracted keypoints are most
probably not going to correspond to the keypoints found in the model in most
cases.

###  5.2 Influences of the Histogram Distance

We observe, that the use of the chi-square histogram distance yields the best
results in the set of experiments on the models superimposed with random noise
(see Section 4.1). However, due to the asymmetry of the particular histogram
distance used, these results should be viewed with caution. The asymmetry
could lead to unbalanced assessments of the similarity of histograms. Using
the symmetrical form of the chi-square histogram distance [5] may yield more
meaningful results in the context of this study. In the experiments with the
TUM-FAÇADE dataset, the use of the Jensen-Shannon divergence leads to the best
results. This histogram, distance is, in contrast to the Chi-Square distance
that we use, symmetric.

###  5.3 Dense feature sampling

We find that using dense feature samples does not improve the performance of
our method compared to the other methods. We attribute this to the binary
properties of the images that we generate from the projected point clouds.
Figure 4 a) demonstrates that almost all important interest points that
characterize the object in the image are identified by the ORB keypoint
detector. When using dense feature sampling, most points for which the
descriptors are determined are located in empty regions of our binary images
or in the vicinity of straight lines. Therefore the information gain of using
dense feature sampling is not particularly large in context of our method.

##  6 Conclusion

In this paper we present a method for reconstructing façade details by using
MLS point clouds and the BoW concept. We incorporate semi-global features to
address the issue of insufficient distinct features in shapes. In our two sets
of experiments, we demonstrate improved performance compared to the
conventional BoW approach. Our method seems to be sensitive to noise and point
cloud sparsity. Future work could focus at increasing the accuracy of the
method as well as its computational efficiency. In the Future, our method
could be applied in façade reconstruction pipelines that aim at a more
realistic reconstruction without assumptions of rectangularity or the use of
bounding boxes.

##  7 Acknowledgements

This work was supported by the Bavarian State Ministry for Economic Affairs,
Regional Development and Energy within the framework of the IuK Bayern project
MoFa3D - Mobile Erfassung von Fassaden mittels 3D Punktwolken, Grant No.
IUK643/001. Moreover, the work was conducted within the framework of the
Leonhard Obermeyer Center at the Technical University of Munich (TUM). We
gratefully acknowledge the Geoinformatics team at the TUM for the valuable
insights and for providing the CityGML datasets.

## References

  * [1] Bamler, R., Shi, Y.: Estimation theory: 2 - Recap of basic propability theory -part 4. 13-5-2022, Chair of Remote Sensing Technology Technical University of Munich. Lecture of Estimation Theory
  * [2] Biljecki, F., Stoter, J., Ledoux, H., Zlatanova, S., Çöltekin, A.: Applications of 3D city models: State of the art review. ISPRS International Journal of Geo-Information 4(4), 2842–2889 (2015)
  * [3] Bronstein, A.M., Bronstein, M.M., Guibas, L.J., Ovsjanikov, M.: Shape google. ACM Transactions on Graphics 30(1), 1–20 (2011). https://doi.org/10.1145/1899404.1899405
  * [4] Calonder, M., Lepetit, V., Ozuysal, M., Trzcinski, T., Strecha, C., Fua, P.: BRIEF: Computing a local binary descriptor very fast. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(7), 1281–1298 (2012). https://doi.org/10.1109/tpami.2011.222
  * [5] Cha, S.H.: Taxonomy of nominal type histogram distance measures. In: Long, C., Sohrab, S.H., Bognar, G., Perlovsky, L. (eds.) MATH’08: Proceedings of the American Conference on Applied Mathematics. pp. 325–330. World Scientific and Engineering Academy and Society (2008)
  * [6] Congalton, R.G., Green, K.: Assessing the accuracy of remotely sensed data: Principles and practices. Taylor & Francis, 3 edn. (2019)
  * [7] Csurka, G., Dance, C.R., Fan, L., Willamowski, J., Bray, C.: Visual categorization with bags of keypoints. In: Proceedings of the ECCV International Workshop on Statistical Learning in Computer Vision (2004)
  * [8] Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). IEEE (2005). https://doi.org/10.1109/cvpr.2005.177
  * [9] Fan, H., Kong, G., Zhang, C.: An interactive platform for low-cost 3d building modeling from VGI data using convolutional neural network. Big Earth Data 5(1), 49–65 (2021). https://doi.org/10.1080/20964471.2021.1886391
  * [10] Fröch, T.: Project photogrammetry - Reconstructing façade details using MLS point clouds and bag-of-words approach. Tech. rep., Technical University Munich (2023), https://github.com/ThomasFroech/ReconstructingFacadeDetailsBoW
  * [11] Grandini, M., Bagli, E., Visani, G.: Metrics for multi-class classification: An overview (2020). https://doi.org/10.48550/ARXIV.2008.05756
  * [12] Hensel, S., Goebbels, S., Kada, M.: Facade reconstruction for textured LOD2 CityGML models based on deep learning and mixed integer linear programming. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences IV-2/W5, 37–44 (2019). https://doi.org/10.5194/isprs-annals-iv-2-w5-37-2019
  * [13] Hoegner, L., Gleixner, G.: Automatic extraction of facades and windows from MLS point clouds using voxelspace and visibility analysis. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLIII-B2-2022, 387–394 (2022). https://doi.org/10.5194/isprs-archives-xliii-b2-2022-387-2022
  * [14] Huang, H., Michelini, M., Schmitz, M., Roth, L., Mayer, H.: LoD3 building reconstruction from multi-source images. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLIII-B2-2020, 427–434 (2020). https://doi.org/10.5194/isprs-archives-XLIII-B2-2020-427-2020, https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLIII-B2-2020/427/2020/
  * [15] Kang, Z., Yang, J.: A probabilistic graphical model for the classification of mobile LiDAR point clouds. ISPRS Journal of Photogrammetry and Remote Sensing 143, 108–123 (2018). https://doi.org/10.1016/j.isprsjprs.2018.04.018
  * [16] Li, J., Lee, G.H.: USIP: Unsupervised stable interest point detection from 3D point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 361–370 (2019)
  * [17] Lowe, D.: Object recognition from local scale-invariant features. In: Proceedings of the Seventh IEEE International Conference on Computer Vision. IEEE (1999). https://doi.org/10.1109/iccv.1999.790410
  * [18] Mallick, S.: Histogram of oriented gradients explained using OpenCV. learnopencv.com (2016), https://learnopencv.com/histogram-of-oriented-gradients/
  * [19] Memon, S.A., Akthar, F., Mahmood, T., Azeem, M., Shaukat, Z.: 3d shape retrieval using bag of word approaches. In: 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET). IEEE (2019). https://doi.org/10.1109/icomet.2019.8673397
  * [20] Nowak, E., Jurie, F., Triggs, B.: Sampling strategies for bag-of-features image classification. In: Leonardis, A., Bischof, H., Prinz, A. (eds.) European Conference on Computer Vision 2006, Part IV, Lecture Notes in Computer Science. vol. 3954, pp. 490–503. Springer-Verlag Berlin Heidelberg (2008)
  * [21] Pantoja-Rosero, B.G., Achanta, R., Kozinski, M., Fua, P., Perez-Cruz, F., Beyer, K.: Generating LoD3 building models from structure-from-motion and semantic segmentation. Automation in Construction 141, 104430 (2022)
  * [22] Redmon, J., Farhadi, A.: Yolov3: An incremental improvement (2018). https://doi.org/10.48550/ARXIV.1804.02767
  * [23] Ripperda, N.: Rekonstruktion von Fassadenstrukturen mittels formaler Grammatiken und Reversible Jump Markov Chain Monte Carlo Sampling. Ph.D. thesis, Leibnitz Universität Hannover, Hannover (2010), wissenschaftliche Arbeiten der Fachrichtung Geodäsie und Geoinformatik der Leibniz Universität Hannover
  * [24] Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: An efficient alternative to SIFT or SURF. In: 2011 International Conference on Computer Vision. IEEE (2011). https://doi.org/10.1109/iccv.2011.6126544
  * [25] Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill, Inc., New York, NY, USA (1986)
  * [26] Sipiran, I., Bustos, B.: Harris 3d: a robust extension of the harris operator for interest point detection on 3d meshes. The Visual Computer 27(11), 963–976 (2011). https://doi.org/10.1007/s00371-011-0610-y
  * [27] Trimble Inc.: SketchUp 3d warehouse (2021), https://3dwarehouse.sketchup.com
  * [28] Tuttas, S., Stilla, U.: Reconstruction of façades in point clouds from multi aspect oblique ALS. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences II-3/W3, 91–96 (Oct 2013). https://doi.org/10.5194/isprsannals-II-3-W3-91-2013, http://www.isprs-ann-photogramm-remote-sens-spatial-inf-sci.net/II-3-W3/91/2013/
  * [29] Wysocki, O., Grilli, E., Hoegner, L., Stilla, U.: Combining visibility analysis and deep learning for refinement of semantic 3d building models by conflict classification. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences X-4/W2, 289–296 (2023). https://doi.org/10.5194/isprs-annals-x-4-w2-2022-289-2022
  * [30] Wysocki, O., Hoegner, L., Stilla, U.: TUM-FAÇADE: Reviewing and enriching point cloud benchmarks for façade segmentation. arXiv preprint arXiv:2304.07140 (2023)
  * [31] Xu, S., Fang, T., Li, D., Wang, S.: Object classification of aerial images with bag-of-visual words. IEEE Geoscience and Remote Sensing Letters 7(2), 366–370 (2010). https://doi.org/10.1109/lgrs.2009.2035644
  * [32] Zhu, Q., Zhong, Y., Zhao, B., Xia, G.S., Zhang, L.: Bag-of-visual-words scene classifier with local and global features for high spatial resolution remote sensing imagery. IEEE Geoscience and Remote Sensing Letters 13(6), 747–751 (2016). https://doi.org/10.1109/lgrs.2015.2513443