# Improving Citation Text Generation: Overcoming Limitations in Length Control

Biswadip Mandal, Xiangci Li, Jessica Ouyang
The university of Texas at Dallas, Richardson, TX, USA
{biswadip.mandal, xiangci.li<EMAIL_ADDRESS>
###### Abstract

A key challenge in citation text generation is that the length of generated
text often differs from the length of the target, lowering the quality of the
generation. While prior works have investigated length-controlled generation,
their effectiveness depends on knowing the appropriate generation length. In
this work, we present an in-depth study of the limitations of predicting
scientific citation text length and explore the use of heuristic estimates of
desired length.

Improving Citation Text Generation: Overcoming Limitations in Length Control

Biswadip Mandal, Xiangci Li, Jessica Ouyang The university of Texas at Dallas,
Richardson, TX, USA {biswadip.mandal, xiangci.li<EMAIL_ADDRESS>

##  1 Introduction

Citation text generation for scientific articles is the task of summarizing
the content of a cited paper as it relates to the citing paper. One key
challenge is that the lengths of human-written citations vary greatly. For
example, if the citing paper directly builds on the approach of the cited
paper, the citation text may be longer and more detailed; if the citing paper
mentions the cited paper as one among many similar background works, the
citation may be shorter. The challenge for the generation model is to produce
a citation that is neither too long, containing irrelevant or redundant
information, nor too short, lacking sufficient information.

In this work, we use the dataset of Li et al. (2022) and use their annotated
citation spans as our generation targets. We observe that many of the
citations produced by Li et al.’s length-agnostic generation model are of a
different length than the human-written targets. Figure 1 shows two example
citations generated by Li et al.’s model, and Figure 2 shows a histogram of
length differences between target citations and Li et al.’s output.

Target: Parvez et al. (2018) explicitly model the type of the next word in
addition to the word itself. Baseline: Parvez et al. (2018) extend vanilla LMs
with a language model to model the compatibility between context sentences and
labels. However, their approach does not explicitly address the problem of
noisy labels. Oracle: Parvez et al. (2018) propose to use LM-enhancement to
improve entity typing. Target: …neural-based methods (Chu et al., 2016…) learn
to identify parallel sentences in the semantic spaces. However, these methods
require large amounts of parallel sentence pairs…which does not apply to
languages with limited resources. Baseline: …proposals (Chu et al., 2016…) use
the corpora to train models and extract parallel sentences. Oracle:
…approaches (Chu et al., 2016…) extract parallel sentences from comparable
monolingual corpora using neural networks. However, these methods are only for
a small number of languages and do not use any external resources such as
dictionaries or corpora.
---
Figure 1: Examples of generated citations that are too long (top) or too short
(bottom). Figure 2: Length difference in tokens between ground truth citations
and Li et al. (2022)’s generated citations.

In this work, we experiment with mitigating this length difference by
predicting and controlling the length of the generated citation. Controlling
generation length in encoder-decoder models has been previously studied in
Kikuchi et al. (2016a); Fan et al. (2017); Liu et al. (2018). However,
predicting the appropriate generation length, rather than using an arbitrary
one (eg. a summary length limit), has received less attention. Additionally,
what work exists on length prediction Yang et al. (2020); Oka et al. (2020)
has been for the task of machine translation, where the lengths of the input
and output are correlated because the two must be highly semantically similar.
In contrast, for citation generation, the input consists of the abstract of
the cited paper and the surrounding context of the target citation, while the
output is a concise summary of the cited paper as it relates to the context;
the length of the output is not necessarily related to that of the input.

We tackle the challenge of generating citations that are both informative and
of appropriate length. Our main contributions are as follows:

  * •

We demonstrate that controlling the length of generated citation produces a
significant improvement in their quality.

  * •

We find that automatic citation length prediction is very difficult, and
heuristic length estimates give better performance on the downstream citation
generation task.

  * •

We provide insights into the behavior of generated citations when length is
modified.

##  2 Related Work

Citation Text Generation. Older, extractive approaches select salient
sentences from cited papers to serve as citations Hoang and Kan (2010); Hu and
Wan (2014); Chen and Zhuge (2019), while recent work focuses on abstractive
approaches to generate citations using neural seq2seq models Xing et al.
(2020); Ge et al. (2021); Luu et al. (2021); Chen et al. (2021); Li et al.
(2022). However, no prior work has investigated the length difference issue
addressed in this work, nor attempted length prediction or control for
citation text generation.

Length Prediction and Control. Many works have studied length control Kikuchi
et al. (2016b); Fan et al. (2018); Liu et al. (2018); Lakew et al. (2019), but
few have addressed length prediction. For machine translation, where the input
and output lengths are related based on their shared semantics, Yang et al.
(2020) trained length prediction and generation jointly, but they used length
prediction only as an auxiliary training task to improve translation. Oka et
al. (2020) used a length prediction model to control a translation model, but
the two were trained separately and used in a pipeline.

##  3 Predicting and Controlling Length

We explore jointly predicting and controlling length for citation generation.
We experiment with a multi-task approach where the predicted length controls
generation, allowing our length regression model to receive additional
training signal from the downstream generation task. We use Longformer
Encoder-Decoder (LED; Beltagy et al., 2020) for generation and the length-
difference position encoding (LDPE) control approach of Takase and Okazaki
(2019), which uses the decoder’s positional encoding to represent the
remaining generation length at each time step (Appendix A.1).

###  3.1 Automatic Length Prediction

Figure 3: Architecture of our joint length prediction and controlled citation
generation models.

Following Li et al. (2022), the input $x$ to our system is the concatenation
of the citing paper’s introduction section; the paragraph containing the
(masked) target citation; and the citation mark (eg. “Smith et al. (2023)"),
title, and abstract of the cited paper. To predict the citation length, we
take the encoding of the $CLS$ meta-token from the LED encoder and pass it
through a feed-forward network ($FF$) to predict a scalar value for length. We
then use the predicted length to control the length of the generated citation,
allowing the length regression model to receive training updates from the
downstream generation model (Equation 1).

$\displaystyle\mathbf{H}$ $\displaystyle=LED_{enc}(\mathbf{x})$ (1)
$\displaystyle\hat{len}$ $\displaystyle=FF(\mathbf{H}_{<CLS>})$
$\displaystyle\hat{y}$
$\displaystyle=LED_{dec}(\mathbf{H}+\mathbf{LDPE}(\hat{len}))$

We explore three approaches for training length regression and controlled
generation (Figure 3).

Vanilla Multitasking. We jointly train length regression and citation
generation using multi-task learning. We pass the predicted length to the
generation model for length control and backpropagate the generation loss
through the length regression module. The training loss is given by:

$L_{all}=\lambda_{g}L_{gen}+(1-\lambda_{g})L_{len}$ (2)

$L_{gen}$ and $L_{len}$ are generation (cross-entropy) loss and regression
(root mean square error) loss, respectively. We tried several values for
$\lambda_{g}$, but as we discuss in Section 5, none worked very well; he
reported results use $\lambda_{g}=0.3$.

Scheduled Sampling Multitasking. We use scheduled sampling Bengio et al.
(2015) to gradually introduce the predicted length to the generation model. We
begin by using the ground truth target length with probability $p$,
initialized to a high value $p_{0}$, and slowly decrease it using exponential
decay:

$p=p_{0}*k^{(epoch+\frac{step+1}{total\\_steps})}$ (3)

$total\\_steps$ is the number of updates per epoch, and $k$ is the decay rate;
we use $p_{0}=0.99$ and $k=0.98$. We do not tune these values, but set them
high in order to use the ground truth length for most of training. As we
discuss in Section 5, the vanilla length regression model is not very
accurate, so this strategy favors using ground truth length.

Teacher Forcing Pipeline We use a hybrid approach of Yang et al. (2020); Oka
et al. (2020), where length regression is trained as a separate auxiliary
task, and its predictions are used only at test time; the generation model is
trained using the ground truth length exclusively.

###  3.2 Heuristic Length Estimates

We also experiment with simple statistical estimates of citation length; we
train a separate length-controlled generation model for each heuristic:

  * •

Average citation length across the training set.

  * •

Citation marks: average length across the training set, partitioned by the
number of citation marks (i.e. cited papers referenced).

  * •

Citing paper average citation length, reflecting the author’s style.

  * •

Random sample from the citation length distribution of the training set.

##  4 Experimental Settings

We use the CORWA dataset of NLP-domain related work sections annotated with
citations; due to the length restriction, we refer the reader to Li et al.
(2022) for details of the dataset.

We compare our experimental approaches to three baselines. First, we train a
length control model using the ground truth target citation lengths; this
Oracle model also uses ground truth lengths at test time, serving as an upper
bound on length-controlled citation generation performance.

Second, the length-agnostic model of Li et al. (2022) serves as a lower bound
that length prediction and control should improve upon. Like our models, Li et
al. use LED Beltagy et al. (2020) with default settings and the same input
format.

Finally, we use one-shot prompting with GPT-3.5 Turbo for length-controlled
generation; the prompt contains a demonstration example, all inputs used by
our models, and the ground truth target citation length. This baseline tests
the feasibility of citation length control by prompting LLMs.

##  5 Results and Analysis

###  5.1 Length-Controlled Generation

Model | R-1 | R-2 | R-L
---|---|---|---
Oracle (target length) | 0.295 | 0.083 | 0.224
Li et al. (2022) | 0.271 | 0.079 | 0.212
GPT-3.5 Turbo | 0.257 | 0.063 | 0.187
Vanilla Multitask | 0.236 | 0.073 | 0.197
Scheduled Sampling | 0.254 | 0.061 | 0.193
Teacher Forcing | 0.269 | 0.068 | 0.199
H. Average | 0.271 | 0.075 | 0.208
H. Citation Marks | 0.276 | 0.077 | 0.212
H. Citing Paper | 0.280 | 0.077 | 0.223
H. Random | 0.241 | 0.065 | 0.187
Table 1: Performance of predicted and heuristic lengths for controlled
generation (ROUGE F1).

Table 1 shows the performance of our multitask approaches to combining length
prediction and control, as well as our length estimate heuristics, compared
with the baselines.

Citation length matters. The Oracle model achieves the best performance, a
significant improvement over Li et al.’s length-agnostic baseline. Further,
the Random heuristic has the lowest performance, even worse than baseline. To
illustrate the importance of generation length, we see in the first example in
Figure 1 that the Target does not criticize the cited paper, and the Baseline
erroneously does so; the Oracle citation fixes this problem by eliminating the
extra sentence. An opposite case can be seen in the second example.

Citation length is idiosyncratic. Of our approaches, the Citing Paper
heuristic performs best, suggesting that citation lengths within a paper do
not vary as much as citations from different papers. This may explain why
length prediction is so challenging; citation length depends on the author.

###  5.2 The Effects of Length Prediction

To examine why joint length prediction and control was outperformed by the
Citing Paper heuristic, we use mean absolute error (MAE) to measure our length
regression performance. We evaluate the ability of the generation model to
control output length using Control Variance Liu et al. (2018):

$Control\\_Var=0.001*\frac{1}{n}\sum_{i=0}^{n}|l_{i}-len_{i}|^{2}$ (4)

$n$ is the number of datapoints and $l_{i}$ and $len_{i}$ are the generated
and desired length, respectively.

Model | MAE | Control Variance
---|---|---
Oracle | - | 0.0001
GPT-3.5 Turbo | - | 0.1852
Vanilla Multitask | 15.04 | 0.5170
Scheduled Sampling | 13.45 | 0.0138
Teacher Forcing | 13.84 | 0.0051
Table 2: Performance on length regression and control. Length regression is
evaluated using MAE in number of tokens; control is evaluated using variance
between desired and generated length.

Citation length prediction is hard. We find that all of our length regression
models struggle to accurately predict citation length. Table 2 shows that all
three approaches have MAE of 13 tokens or more, which is significant,
considering the average length of citations in the dataset is only 34.5
tokens. We do find that the Vanilla and Scheduled Sampling strategies, which
are updated with the generation loss, perform better than Teacher Forcing, but
overall their performance is poor and causes the downstream generation task to
perform worse than Baseline. Further, we see that, while the Oracle approach
exhibits a high level of controllability, our multitask strategies do not.
This gap may arise from our predicted length being so noisy: the downstream
length control model learns to ignore this unreliable length signal.

To further investigate the impact of input length errors, we conduct an
experiment where we take the same target citation and generate it at different
lengths: 20, 30, or 50 tokens. We find that forcing the model to generate
short citations results in vague, generic citations. Increasing the length
results in additional sentences being generated, which can comment about the
methodology, add more information, or criticize the cited paper. However, when
the desired length exceeds the ground truth, the model begins to hallucinate
these comments, especially criticisms. Figure 4 shows a citation being
generated at both the correct (20 tokens) and a longer (50 token) length. The
longer generation hallucinates a criticism that clearly contradicts the cited
paper’s abstract.

Finally, we see that GPT-3.5 has relatively poor length controllability,
corroborating the observations of Sun et al. (2023) regarding the management
of output length in LLMs through prompting.

len=20: …Wang and Lu (2018a) proposed a segmental hypergraph model for
modeling. len=50: …Wang and Lu (2018a) proposed a segmental hypergraph
representation to model overlapping entities. However, their model is not able
to capture the interactions between entities with overlapping spans. Wang and
Lu (2018a) abstract: …We show that our model built on top of such a new
representation is able to capture features and interactions that cannot be
captured by previous models…
---
Figure 4: Example of over-long citation resulting in a hallucinated criticism
of the cited paper.

##  6 Conclusion

We have explored different approaches for predicting the lengths of citations
and using length to control generation. Our experimental results demonstrate
that using the ground truth length can significantly improve the quality of
the generated citations, but predicting the length of a citation based solely
on the cited paper abstract and target citation context is challenging. To
address this issue, we propose the use of heuristic estimates of desired
length, finding that citation length is mostly related to each author’s
individual writing style, which explains why it is so difficult to predict.
Our work highlights an important challenge in citation text generation and
suggests a straightforward solution. By using author-specific citation length
estimates, we can eliminate the length difference between the generated and
ground truth citations, significantly improving the quality of the generated
citations. We have made our code available at
https://github.com/mandalbiswadip/LengthControlledGeneration

## References

  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv:2004.05150_.
  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled sampling for sequence prediction with recurrent neural networks.  _Advances in neural information processing systems_ , 28.
  * Chen and Zhuge (2019) Jingqiang Chen and Hai Zhuge. 2019.  Automatic generation of related work through summarizing citations.  _Concurrency and Computation: Practice and Experience_ , 31(3):e4261.
  * Chen et al. (2021) Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Xiangliang Zhang, Dongyan Zhao, and Rui Yan. 2021.  Capturing relations between scientific papers: An abstractive model for related work section generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6068–6077, Online. Association for Computational Linguistics.
  * Fan et al. (2017) Angela Fan, David Grangier, and Michael Auli. 2017.  Controllable abstractive summarization.  _arXiv preprint arXiv:1711.05217_.
  * Fan et al. (2018) Angela Fan, David Grangier, and Michael Auli. 2018.  Controllable abstractive summarization.  In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_ , pages 45–54.
  * Ge et al. (2021) Yubin Ge, Ly Dinh, Xiaofeng Liu, Jinsong Su, Ziyao Lu, Ante Wang, and Jana Diesner. 2021.  BACO: A background knowledge- and content-based framework for citing sentence generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1466–1478, Online. Association for Computational Linguistics.
  * Hoang and Kan (2010) Cong Duy Vu Hoang and Min-Yen Kan. 2010.  Towards automated related work summarization.  In _Coling 2010: Posters_ , pages 427–435.
  * Hu and Wan (2014) Yue Hu and Xiaojun Wan. 2014.  Automatic generation of related work sections in scientific papers: An optimization approach.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1624–1633, Doha, Qatar. Association for Computational Linguistics.
  * Kikuchi et al. (2016a) Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016a.  Controlling output length in neural encoder-decoders.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1328–1338, Austin, Texas. Association for Computational Linguistics.
  * Kikuchi et al. (2016b) Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016b.  Controlling output length in neural encoder-decoders.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1328–1338.
  * Lakew et al. (2019) Surafel Melaku Lakew, Mattia Di Gangi, and Marcello Federico. 2019.  Controlling the output length of neural machine translation.  In _Proceedings of the 16th International Conference on Spoken Language Translation_.
  * Li et al. (2022) Xiangci Li, Biswadip Mandal, and Jessica Ouyang. 2022.  CORWA: A citation-oriented related work annotation dataset.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5426–5440, Seattle, United States. Association for Computational Linguistics.
  * Liu et al. (2018) Yizhu Liu, Zhiyi Luo, and Kenny Zhu. 2018.  Controlling length in abstractive summarization using a convolutional neural network.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4110–4119, Brussels, Belgium. Association for Computational Linguistics.
  * Luu et al. (2021) Kelvin Luu, Xinyi Wu, Rik Koncel-Kedziorski, Kyle Lo, Isabel Cachola, and Noah A. Smith. 2021.  Explaining relationships between scientific documents.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2130–2144, Online. Association for Computational Linguistics.
  * Oka et al. (2020) Yui Oka, Katsuki Chousa, Katsuhito Sudoh, and Satoshi Nakamura. 2020.  Incorporating noisy length constraints into transformer with length-aware positional encodings.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 3580–3585.
  * Sun et al. (2023) Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Frederick Wieting, Nanyun Peng, and Xuezhe Ma. 2023.  Evaluating large language models on controlled generation tasks.  _arXiv preprint arXiv:2310.14542_.
  * Takase and Okazaki (2019) Sho Takase and Naoaki Okazaki. 2019.  Positional encoding to control output sequence length.  _arXiv preprint arXiv:1904.07418_.
  * Xing et al. (2020) Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020.  Automatic generation of citation texts in scholarly papers: A pilot study.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6181–6190.
  * Yang et al. (2020) Zijian Yang, Yingbo Gao, Weiyue Wang, and Hermann Ney. 2020.  Predicting and using target length in neural machine translation.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 389–395, Suzhou, China. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Length-Difference Positional Encoding (LDPE)

Takase and Okazaki (2019) define the LDPE for a Transformer-based encoder-
decoder as follows:

$\displaystyle LDPE(pos,len,2i)$ $\displaystyle=\sin{(\frac{(len-
pos)}{10000^{\frac{2i}{d}}})}$ (5) $\displaystyle LDPE(pos,len,2i+1)$
$\displaystyle=\cos{(\frac{(len-pos)}{10000^{\frac{2i}{d}}})}$

$len$ refers to the desired length of the generation, measured in the number
of tokens.

###  A.2 More Citation Length Examples

Figure 5 shows an example of a citation that is vague and too generic when
generated at a shorter length. Figure 6 shows two examples of citations
generated at shorter or longer lengths, resulting in lower fluency.

len=20: Peng et al. (2018) improve the state-of-the-art. len=30: Peng et al.
(2018) propose a joint model for frame-semantic parsing and semantic
dependency parsing by treating annotations as variables.
---
Figure 5: Example of an overly-short, generic citation. Too short: Alkhouli et
al. (2018) proposed to use an additional alignment head to improve NMT
performance, but their scalability. Too long: Firat et al. (2016b) proposed a
distillation-based distillation approach, in which the distillation process is
trained on a small set of labeled data.
---
Figure 6: Examples of shorter and longer generated citations (compared to the
ground truth target length) resulting in poor fluency.

###  A.3 GPT-3.5 Turbo Prompt

Figure 7 shows the prompt we used for our GPT-3.5 Turbo length control
baseline. The prompt consists of the citation context and a list of papers to
cite, as well as the ground truth target citation length.

We have written an incomplete related work section. We want a set of cited
papers to be summarized in the context of the imcomplete related work section.
Given the incomplete related work section and a list of cited papers, complete
it by summarizing the cited papers in the related work section context in
<Length> words. The position where the summarized text goes is indicated by
the special token ‘[Dominant]’. The cited paper can be one of two types,
referred to as Citation Type: Dominant: These citations are discussed in
detail, usually via summarization of their content, and are often longer than
reference citations. Reference: These citations are not discussed in detail.
Reference citations tend to be more abstract than dominant citations. Output
format: Summarize the cited papers given their Citation Mark, Citation Type,
Title, and Abstract. Return only the piece of text required to complete the
related work section. Return it in string format. The number of words in the
output should be the same as <Length>. Incomplete Related work section:
<Example 1 incomplete related work section> List of cited papers: 1. Citation
mark: <Example 1 first cited paper citation mark> Citation Type: <Example 1
first cited paper citation type> Title: <Example 1 first cited paper title>
Abstract: <Example 1 first cited paper abstract> 2. … Number of output words:
<Example 1 Length> Output: “‘<Example 1 output>”’ Explanation: The generated
output has <Example 1 Length> words, which starts with the word <Example 1
output starting word> and ends with the word <Example 1 output ending word>
Incomplete Related work section: <Input incomplete related work section> List
of cited papers: 1. Citation mark: <Input first cited paper citation mark>
Citation Type: <Input first cited paper citation type> Title: <Input first
cited paper title> Abstract: <Input first cited paper abstract> 2. … Number of
output words: <Length> Output:
---
Figure 7: One-shot prompt used for length-controlled citation generation using
GPT-3.5 Turbo.


# Building Change Detection using Multi-Temporal Airborne LiDAR Data

Ritu Yadav  Andrea Nascetti  Yifang Ban Corresponding author Division of
Geoinformatics, KTH Royal Institute of Technology, Sweden - (rituy, nascetti,
<EMAIL_ADDRESS>
###### Abstract

Building change detection is essential for monitoring urbanization, disaster
assessment, urban planning and frequently updating the maps. 3D structure
information from airborne light detection and ranging (LiDAR) is very
effective for detecting urban changes. But the 3D point cloud from airborne
LiDAR(ALS) holds an enormous amount of unordered and irregularly sparse
information. Handling such data is tricky and consumes large memory for
processing. Most of this information is not necessary when we are looking for
a particular type of urban change. In this study, we propose an automatic
method that reduces the 3D point clouds into a much smaller representation
without losing necessary information required for detecting Building changes.
The method utilizes the Deep Learning(DL) model U-Net for segmenting the
buildings from the background. Produced segmentation maps are then processed
further for detecting changes and the results are refined using morphological
methods. For the change detection task, we used multi-temporal airborne LiDAR
data. The data is acquired over Stockholm in the years 2017 and 2019. The
changes in buildings are classified into four types: ‘newly built’,
‘demolished’, ‘taller’ and ’shorter’. The detected changes are visualized in
one map for better interpretation.

###### keywords:

LiDAR, ALS, U-Net, Segmentation, Change Detection, Urbanization, Remote
Sensing.

##  1 Introduction

With the population growth and rapid development, it became crucial to monitor
uncontrolled urbanization. The rate of people moving from villages to cities
is increasing at a threatening level. Urban changes can be due to the
construction of new railway tracks, roads, construction of the new residential
area, demolition of old or illegal properties, filling lakes, and many others.
For sustainable urban planning, regular updating of footprints and maps is
becoming challenging every year. In such a scenario, remote change detection
can play a significant role in monitoring and understanding the local and
global changes. Change detection can also facilitate environmental monitoring,
resource exploration, and disaster assessment. Conventionally, the urban
changes were detected using methods developed on multi-temporal remote sensed
multi-spectral or optical images. The high-resolution optical data is useful
in change detection but is dependent on sunlight, fog, and multiple other
environmental conditions. On the other hand, LiDAR and Radar active remote
sensors are immune to these factors and better at detecting urban changes.
Radar data is difficult to process and involves a lot of noise issues. In this
scenario, LiDAR data is an effective alternative. LiDAR provides 3D data,
which helps in monitoring volumetric changes as well as 2D changes. ALS has
dramatically changed the monitoring and 3D modeling of districts, cities, etc.
LiDAR provides information such as elevation, intensity, and ’number of
returns’, which helps in segregating different features in the scene. These
attributes are significant in detecting buildings.

In previous studies, ALS data has shown great potential in detecting changes
in buildings. The methods are further explored in combination with aerial
images. One example of such work is [Matikainen et al., 2010], where a
combination of aerial and ALS data is used to detec the buildings. For
detecting buildings, elevation information from ALS is used and arial images
are used to segregate trees from buildings. Changes in buildings are detected
using object based matching algorithm. In [Murakami et al., 1999] study,
multi-temporal high resolution DSM is prepared from the ALS data and changes
are detected by calculating difference between the DSM. Authors of study [Teo
and Shih, 2013], also explored the multi-temporal DSM prepared from ALS data.
DSM are first classified on the basis of surface roughness and the changes in
buildings is detected by change in the land cover class. In [Vu et al., 2004]
study, authors proposed a building change detection method on LiDAR data in
highly dense urban area. Here high resolution ALS grid data is used instead of
point data. Building changes are detected by manually thresholding the
difference histogram of the two grids. In [dos Santos et al., 2020] study also
authors subtract two DSM from the two time stamps and refined the changes
using a height entropy concept.

Figure 1: Data Sample Visualization. From left to right, we have RGB image
which is the orthophoto projection on LiDAR point cloud, second is Ground
truth used for the Building Segmentation task. First row is showing data from
year 2017 and second from 2019.

This work is conducted on multi-temporal ALS data to detect urban changes over
the Stockholm area. In the proposed change detection method we first use DL
segmentation methods to detect all buildings and then proceed with the change
detection process. Pointnet [Qi et al., 2017a] and pointnet++ [Qi et al.,
2017b] are the two recent segmentation methods popular for there state of the
art performance on 3D point cloud data. These DL methods work on consistent
LiDAR data but ALS suffers from the problem of random irregular data issues.
In our ALS dataset, For the majority of sites, LiDAR data points are available
only on the building rooftop. Since, 3D structure information is missing
randomly across the data, using 3D point-cloud segmentation methods will cause
false change detection in the later process. With this knowledge, we decided
to drop the idea of using 3D point cloud segmentation and figure out a way to
segment building in 2D keeping all important features intact.

In this study, we propose one single stream segmentation network which works
on selected LiDAR attributes. The same network can be trained on a
segmentation task for sparse optical data associated with LiDAR data points.
We also presented one dual-stream segmentation architecture where selected
LiDAR features and optical features are fused to get more accurate
segmentation predictions. Before calculating changes between the two-year
buildings, we complemented our segmentation results with interpolation and
morphology-based methods. The presented method attempts to perform the task in
multiple steps, which are namely data processing, 2D projection, segmentation,
building map preparation, area and elevation-wise change detection of
buildings, refining the change maps using morphological algorithms, final
change map visualization, and qualitative assessment of the results.

##  2 Study Area and Data

In this study, the dataset used is recorded over multiple sites in Stockholm
during April 2017 and April 2019. The data is private and belongs to the
Stockholm city government. The dataset is 3D point cloud data recorded using
LiDAR airborne scanners with the point density of 12 points per meter square
ground area.

In total, 27 different sites are scanned and for each site, we have two points
clouds from the years 2017 and 2019. Every point cloud files contain 6 million
to 35 million points. Each point is associated with 12 different information,
such as point coordinates, RGB values, intensity, elevation, number of
returns, scan angle rank, GPS time, and point source id. For the change
detection task explained in the following section, we extracted the point
coordinates and corresponding six attribute values R, G, B, elevation,
intensity, and ’number of returns’. In the dataset, Building size varies from
60 meter square to 1700 meter square. Also, Ground surfaces are relatively
continuous and flat. For a better understanding of the area, one sample from
both 2017 and 2019 is visualized in Figure1.

There are few black sites in the dataset, where data is not available. In
multiple areas, LiDAR data points are sparse and missing. RGB color
information is collected by projecting the Orthophoto of each area on to
corresponding 3D point cloud. Hence the RGB information in the dataset is also
affected by the black thin line and other Lidar point cloud issues. The most
frequent case is missing data on top of the building, creating small holes.
These data points are missing inconsistently between the two-year data and
cause false change detection. Since, the changes are really small in area, to
a large extent we handled this using interpolation.

##  3 Method

The complete workflow of the building change detection is illustrated in
Figure 2 and the steps involved are explained in the following subsections.

Figure 2: Building Change Detection Workflow.

###  3.1 Pre-Processing

Handling a 3D point cloud containing millions of points is complicated to
process as well as heavy on memory. Due to random order of missing 3D building
structure information, it is difficult to maintain the data consistency
between the two years. Detecting changes using inconsistent data will cause
false detection. For detecting building changes, it is not necessary to
utilize all information associated with each point in the point cloud. The
idea is to extract necessary information from the 3D point cloud and then
perform the change detection task. Since we are interested in detecting
spatial changes and the changes in elevations, the top view from the point
cloud is sufficient. For the top view, we need to gather the surface data from
the point cloud, which is done by extracting the surface point for each pixel.
The surface point at a given location is the data point with the maximum
elevation value. The extracted surface data is then projected on a 2D plane.

From the data, a 3-channel matrix is prepared, where each channel is the 2D
projection of the 3D point cloud. The first channel is holding the ’elevation’
information. The second and third channel contains ’intensity’ and ’number of
returns’ data respectively. Both 2017 and 2019 point cloud is processed in the
same way. Similarly, we extract the RGB values of the 2D projected points and
place them in 3 channels.

###  3.2 Single Stream Segmentation Network

For the segmentation task, lightweight U-net [Ronneberger et al., 2015]
architecture is proposed. U-Net is one of the widely used CNN architectures
originally designed for the pixel-level classification of medical images. This
network consists of an encoder-decoder architecture. The encoder extracts the
salient features from the input using several convolutional blocks, max-
pooling, and batch normalization. The output of the encoder is a smaller set
of feature maps. These small features are decoded back to the bigger feature
maps guided to have a limited number of classes. The output is a segmentation
map, where each pixel is classified.

Conventionally, Resnet [He et al., 2016] is the encoder in U-net architecture.
In our proposed network, we replaced resnet backbone with a much smaller
network known as Efficientnet-B4 [Tan and Le, 2019]. In addition to this, we
only used 3 levels of the encoder, unlike most U-net architectures. In the
decoder network, the encoded feature maps are up-sampled using a series of
transpose convolution, convolutional layer, and batch normalization layers.
Overall our proposed network settings contain one-fourth of the parameter in
comparison to resnet based U-net, which makes it significantly lightweight.

The network takes The 3 channel input(ZIN) containing elevation(Z),
intensity(I), and the number of returns(N) information. The output of the
network is a segmentation map with two classes named building and background.
This is a single stream lightweight U-net architecture. For better
understanding, the network is represented using a block diagram in Figure 3.

Figure 3: Single Stream Segmentation Network.

###  3.3 Dual Stream Fusion Segmentation Network

As mentioned in subsection3.1, we have the RGB values corresponding to all the
LiDAR data points. We wanted to compliment our network with optical
information. To achieve that, we propose a dual-stream segmentation network
where the network takes two 3-channel input streams, one with RGB values and
the second(ZIN) stacking elevation, intensity, and the number of returns. The
inputs from the two streams are encoded separately using two Efficientnet-B4
encoders. The encoded features are then fused and supplied to the decoder
network to prepare a binary segmentation map. The encodes feature maps from
the two streams are fused using a concatenation operation. The block diagram
of our dual-stream segmentation network is presented in Figure 4.

Figure 4: RGB and LiDAR Fused Dual Stream Segmentation Network. Figure 5:
Segmentation Result Samples. From left to right, we have RGB image, ZIN
3-channel representation, ground truth, prediction from the Single Stream
Segmentation Network on RGB input, prediction from the Single Stream
Segmentation Network on ZIN input and prediction from the Dual Stream Fusion
Segmentation Network.

These segmentation networks are trained on input patches of size 128x128. The
LiDAR samples in our data are of different dimensions. These samples are
divided into small patches of size 128x128 and fed into the network with a
batch size of 4. We also applied a list of data augmentation to the input
patches. Some of the augmentations are horizontal, vertical flips, rotations,
crop, pad, translate. The model is trained for 150 epochs using binary cross-
entropy loss and Adam optimizer. The learning rate is set to 0.001 along with
the decay rate of 0.1 after every 20 steady epochs. These hyperparameter
settings are applied for all experiments. We implemented everything in python
using Keras framework and the training is conducted on a single Google colab
GPU.

###  3.4 Change Detection

For change detection, a 3D matrix with 2-channels is prepared. The first
channel is the segmentation prediction from the trained model. The second
channel contains elevation values corresponding to the points which are
predicted as buildings by our model. This 3D matrix data is prepared for both
the 2017 and 2019 samples covering the same area. Now, the channel-wise
difference is calculated between the two-year 3D matrix. This difference is
then categorized into positive and negative changes.

The difference in the first channel represents the changes in area or the
changes from the ground. The positive change here can be the construction of a
new building in the year 2019 and this area used to be either empty or under a
different class in 2017. The negative changes can be due to the demolishment
of a building after recording the first data samples in 2017. The difference
in the second channel is only about changes in elevation i.e building getting
taller or shorter. This covers the scenarios, where a particular area was
classified as building in 2017 as well as in 2019. But, in one of the years,
it was a construction site and in another year the building height has
increased by some floors or decreased(i.e while reconstructing the top floor,
partially damaged in some disaster event).

Morphological operations are applied to reduce the noise in the change
results. The operations used are ’closing’ and ’opening’ with the kernel size
3. Once the results are refined, the change maps for each category are
prepared. Both maps consist of negative and positive changes. For better
interpretation, the two change maps are overlaid with the building footprint
from 2019. These final change maps are further explained in the result
section.

##  4 Results and Evaluation

First, we present the quantitative and qualitative comparison of segmentation
methods described in the method section. The best segmentation method is
chosen to conduct the change detection task. In the later section, a
qualitative analysis of the change detection results is shown and discussed in
detail.

###  4.1 Segmentation Evaluation

Network performance is assessed in terms of IOU score taking into account True
Positives (TP), False Positives (FP) and False Negatives (FN). The IOU score
is described in equation 1.

$IOU=\frac{TP}{TP+FP+FN}$ (1)

Figure 6: Change Detection Result Samples.  Figure 7: Negative Change
Detection sample.

For the quantitative comparison, Table 1 list IOU scores of the three methods;
single stream segmentation on ZIN, single stream segmentation on RGB and the
dual stream fusion network on both RGB and ZIN. The single stream network
trained on ZIN input achieved higher IOU score than the one trained on RGB
input. The 10% performance difference is considerably high. The performance is
enhanced further by approximately by 2%, when both RGB and ZIN inputs are
fused in training the dual stream fusion network.

The qualitative comparison of the three segmentation methods is visualized in
Figure 5. The visualized samples show agreement with the quantitative results
and give more insight into specific differences. The performance of the RGB
input-based single stream network is substantially low in comparison to
others. This can be visually demonstrated by the prediction samples shown in
the second row of Figure 5. In the first row, a small area is highlighted
using a dashed circle. According to the label, there are four small buildings
with low elevations. The single-stream network detected either one building
when trained on ZIN data or none when trained on RGB data. Whereas, the fusion
network successfully detected three small buildings. The possible explanation
here is that the RGB adds specific building color information to the network.
Elevation, intensity, and color information together provide better features
to the fusion network, which help in recognizing small buildings.

Our proposed fusion network scored 86.7% IOU. We analyzed the remaining 13% of
the building which are not detected by our network. These are the small
buildings with low elevations. Network struggles more if these small buildings
are partially covered by dense trees. These small buildings or partly not
visible in optical images. With current point density the intensity and
’number of returns’ also fail to differentiate small buildings from other.

Methods | IOU
---|---
RGB Single Stream | 75
ZIN Single Stream | 85
RGB+ZIN Dual Stream | 86.7

Table 1: Performance Comparison of Segmentation Methods.

###  4.2 Change Detection Evaluation

Two samples are shown in Figure 6 for qualitative analysis of the change
detection. The first row of the figure contains the data from the year 2017
and the second row is for 2019 data. From left to right we have: First the RGB
visualization of the area, which is not an orthophoto but the RGB information
corresponding to the points in 2D surface data. The second is the
visualization of the building footprints. The buildings are in blue color and
the background is in black. In the third column holds the two change maps. The
first change map contains changes corresponding to newly built(positive
changes) and demolished buildings(negative changes). The new buildings are in
white and demolished buildings are in black. It is crucial to note that this
map indicates the area-wise changes along with elevation changes. The second
change map contains only elevation changes. These changes exclude all
elevation changes pointed by the first change map and focused on increase or
decrease in number of floors. The fourth is the visualization of the final
change map, where all the changes are projected onto the 2019 building
footprint. In the final change map, buildings are in blue, (newly built and
demolished buildings) are in red and elevation changes are in off-white.

Following the presented workflow, we have prepared and analyzed the change
maps for all the sites in the dataset. The entire process is automatic and can
be performed end to end. The detected changes were confirmed by the officials
along with comments on false building detections which are are mainly the
sheds installed on top of the building during construction. These sheds give
an impression of extra height to the building and these elevation differences
are falsely detected as building changes. One of such examples is shown in
Figure 7. From left to right, we presented RGB visualization of samples from
2017 and 2019 in the first and second row respectively. The second column is
the 3D visualization of the point cloud where 2017 data is in Green color and
2019 data is in blue. The problem area is highlighted with a white circle. The
third column is the change map, where elevation changes are detected in the
highlighted building. The detected change is false positive and happened due
to shed installment during the reconstruction of the top floors of the
building. It is noteworthy that these false positives are still an indicator
of ongoing construction on top of the building and can be utilized for
monitoring purposes.

##  5 Conclusion

In this paper, we presented a an approach to handle the irregularity of 3D
point cloud data from ALS and utilize efficiently for building change
detection. Our method explores the benefit of several LiDAR attributes in 2D
which are extracted from the 3D point cloud. We proposed two lightweight
networks for segmenting buildings from the background and presented the
performance comparison. First architecture operates on single modality either
optical or ALS whereas the second architecture benefits from both optical and
LiDAR features.

Our findings indicate that the segmentation results are better when both
optical and ALS dat are fused. Our change detection approach generate three
change maps explained in the previous section. All change maps are
georeferenced and can be visualized in tools like QGIS, ArcGIS, GEE, and
CloudCompare. These change maps and quick visualization can help in enhancing
the monitoring of construction of new buildings, horizontal extension of
building such as illegal balcony extension if visible in top view,
demolishment of old buildings, completion/ongoing construction, partial or
complete building damage due to disaster events and many others.

## References

  * [dos Santos et al., 2020] dos Santos, R., Galo, M., Carrilho, A., Pessoa, G., de Oliveira, R., 2020. Automatic building change detection using multi-temporal airborne lidar data. _2020 IEEE Latin American GRSS & ISPRS Remote Sensing Conference (LAGIRS)_, IEEE, 54–59.
  * [He et al., 2016] He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * [Matikainen et al., 2010] Matikainen, L., Hyyppä, J., Ahokas, E., Markelin, L., Kaartinen, H., 2010. Automatic detection of buildings and changes in buildings for updating of maps. Remote Sensing, 2(5), 1217–1248.
  * [Murakami et al., 1999] Murakami, H., Nakagawa, K., Hasegawa, H., Shibata, T., Iwanami, E., 1999. Change detection of buildings using an airborne laser scanner. ISPRS Journal of Photogrammetry and Remote Sensing, 54(2-3), 148–152.
  * [Qi et al., 2017a] Qi, C. R., Su, H., Mo, K., Guibas, L. J., 2017a. Pointnet: Deep learning on point sets for 3d classification and segmentation. _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 652–660.
  * [Qi et al., 2017b] Qi, C. R., Yi, L., Su, H., Guibas, L. J., 2017b. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413.
  * [Ronneberger et al., 2015] Ronneberger, O., Fischer, P., Brox, T., 2015. U-net: Convolutional networks for biomedical image segmentation. _International Conference on Medical image computing and computer-assisted intervention_ , Springer, 234–241.
  * [Tan and Le, 2019] Tan, M., Le, Q., 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. _International Conference on Machine Learning_ , PMLR, 6105–6114.
  * [Teo and Shih, 2013] Teo, T.-A., Shih, T.-Y., 2013. Lidar-based change detection and change-type determination in urban areas. International Journal of Remote Sensing, 34(3), 968–981.
  * [Vu et al., 2004] Vu, T. T., Matsuoka, M., Yamazaki, F., 2004. Lidar-based change detection of buildings in dense urban areas. _IGARSS 2004. 2004 IEEE International Geoscience and Remote Sensing Symposium_ , 5, IEEE, 3413–3416.