Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

CDCEO 2022: 2nd Workshop on Complex Data Challenges in Earth Observation, July
25, 2022, Vienna, Austria

<EMAIL_ADDRESS>] [1] [1]

<EMAIL_ADDRESS>] [1]

<EMAIL_ADDRESS>] [1]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

[1]Corresponding author. [1]These authors contributed equally.

# METER-ML: A Multi-Sensor Earth Observation Benchmark for Automated Methane
Source Mapping

Bryan Zhu  Department of Computer Science, Stanford University  Nicholas Lui
Department of Statistics, Stanford University  Jeremy Irvin  Jimmy Le  Sahil
Tadwalkar  Department of Civil and Environmental Engineering, Stanford
University  Chenghao Wang  Zutao Ouyang  Frankie Y. Liu  Department of Earth
System Science, Stanford University  Andrew Y. Ng  Robert B. Jackson  Woods
Institute for the Environment and Precourt Institute for Energy, Stanford
University

(2022)

###### Abstract

Reducing methane emissions is essential for mitigating global warming. To
attribute methane emissions to their sources, a comprehensive dataset of
methane source infrastructure is necessary. Recent advancements with deep
learning on remotely sensed imagery have the potential to identify the
locations and characteristics of methane sources, but there is a substantial
lack of publicly available data to enable machine learning researchers and
practitioners to build automated mapping approaches. To help fill this gap, we
construct a multi-sensor dataset called METER-ML containing 86,599
georeferenced NAIP, Sentinel-1, and Sentinel-2 images in the U.S. labeled for
the presence or absence of methane source facilities including concentrated
animal feeding operations, coal mines, landfills, natural gas processing
plants, oil refineries and petroleum terminals, and wastewater treatment
plants. We experiment with a variety of models that leverage different spatial
resolutions, spatial footprints, image products, and spectral bands. We find
that our best model achieves an area under the precision recall curve of 0.915
for identifying concentrated animal feeding operations and 0.821 for oil
refineries and petroleum terminals on an expert-labeled test set, suggesting
the potential for large-scale mapping. We make METER-ML freely available at
this link to support future work on automated methane source mapping.

###### keywords:

Earth observationremote sensingmachine learning deep learning datasetclimate
changemethane

##  1 Introduction

Figure 1: METER-ML is a multi-sensor dataset containing 86,599 examples of
NAIP aerial imagery, Sentinel-2 satellite imagery, and Sentinel-1 satellite
imagery. We include 19 spectral bands across these three products, with the
RGB and VH&VV bands shown here. Each example is labeled with the presence or
absence of six different methane source facilities and is georeferenced. A
small amount of examples are labeled to contain facilities from more than one
category and 34,870 examples contain no facilities from the six categories.

Anthropogenic methane emissions are the main contributor to the rise of
atmospheric methane [1], and mitigating methane emissions is widely recognized
as crucial for slowing global warming and achieving the goals of the Paris
Agreement [2]. Multiple satellites are in orbit or launching soon which will
measure methane emissions from the surface using top-down approaches, but in
order to attribute these emissions to specific sources on the ground, a
comprehensive database of methane emitting infrastructure is necessary [3].
Although several public databases of this infrastructure exist, the data
available globally is incomplete, erroneous, and unaggregated.

AI approaches on Earth observation data have the potential to fill in this
gap. Several recent works have developed deep learning models to automatically
interpret remotely sensed imagery and deploy them at scale to map
infrastructure [4, 5, 6, 7]. Methods for mapping methane source infrastructure
have been emerging as well, including well pads in the Denver basin [8], oil
refineries and concentrated animal feed operations in the U.S. [9, 10], and
wastewater treatment plants in Germany [11]. Each of these works depended on
the curation of large, labeled datasets to develop the machine learning
models, but there is a lack of publicly available, labeled Earth observation
data, specifically on methane emitting infrastructure, which prohibits
researchers and practitioners from building automated mapping approaches.

In this work, we construct a multi-sensor Earth observation dataset for
methane source infrastructure identification called METER-ML. In support of a
new initiative to build a global database of methane emitting infrastructure
called the MEthane Tracking Emissions Reference (METER) [12], we develop
METER-ML to allow the machine learning community to experiment with multi-
view/multi-modal modeling approaches to automatically identify this
infrastructure in remotely sensed imagery. METER-ML includes georeferenced
imagery from three remotely sensed image products, specifically 19 spectral
bands in total from NAIP, Sentinel-1, and Sentinel-2, capturing 51,729 sources
of methane from six different classes as well as 34,870 negative examples
(Figure 1). The dataset includes expert-reviewed validation and test sets for
robustly evaluating the performance of derived models. Using the dataset, we
experiment with a variety of convolutional neural network models which
leverage different spatial resolutions, spatial footprints, image products,
and spectral bands. The dataset is freely
available111https://zenodo.org/record/6911013 in order to encourage further
work on developing and validating methane source mapping approaches.

##  2 Methods

###  2.1 Methane source locations

We collect locations of methane emitting infrastructure in the U.S. from a
variety of public datasets. We focus on the U.S. in this study due to the high
availability of publicly accessible infrastructure data and remotely sensed
imagery. The infrastructure categories we include are concentrated animal
feeding operations (CAFOs), coal mines (Mines), landfills (Landfills), natural
gas processing plants (Proc Plants), oil refineries and petroleum terminals
(including crude oil and liquified natural gas terminals), and wastewater
treatment plants (WWTPs). We group oil refineries and petroleum terminals
together due to their high similarity in appearance, and refer to that
category as “Refineries & Terminals” (R&Ts). These infrastructure categories
were chosen based on their potential for emitting methane along with their
consistent, visible differentiating features which make them feasible to
identify in high resolution remotely sensed imagery.

The locations are obtained from 18 different publicly available datasets, all
of which have licenses that allow for redistribution (see Table 6 in the
Appendix). As various datasets may contain the same locations of
infrastructure, we deduplicate by considering locations within 500m of each
other identical. In total we include 51,729 unique locations of methane source
infrastructure in the dataset, which we refer to as positive examples.

Table 1: Counts and proportions of each category in METER-ML. The labels on
the training set are obtained from public data whereas the labels on
validation and test sets are obtained from a consensus of two methane source
identification experts. The individual category counts do not add up to the
overall train/valid/test counts as some (0.8%) of the positive examples are
labeled with more than one methane source category.

Category | Train (%) | Valid (%) | Test (%) | Total
---|---|---|---|---
CAFOs | 24957 (29.3%) | 47 (9.1%) | 92 (9.0%) | 25096
Landfills | 4085 (4.8%) | 46 (8.9%) | 111 (10.9%) | 4242
Coal Mines | 1776 (2.1%) | 40 (7.8%) | 72 (7.1%) | 1888
Proc Plants | 1900 (2.2%) | 38 (7.4%) | 107 (10.5%) | 2045
R&Ts | 4012 (4.7%) | 59 (11.5%) | 108 (10.6%) | 4179
WWTPs | 14519 (17.1%) | 46 (8.9%) | 129 (12.7%) | 14694
Negatives | 34195 (40.2%) | 249 (48.3%) | 426 (41.8%) | 34870
Total | 85066 | 515 | 1018 | 86599

###  2.2 Negative locations

We additionally include a variety of images in the dataset which capture none
of the six methane emitting facilities. To do this, we define around 50
classes (see Appendix) of different facilities and landscapes and select
characteristic examples of each class. Then we collect locations containing
similar facilities and landscapes using the Descartes Labs GeoVisual Search
[13], providing up to 1000 similar locations per example. A sample of the
similar locations were manually vetted in each case to ensure no locations
obtained actually corresponded to the six methane source categories. In total
we include 34,870 locations of facilities and landscapes which are not any of
the six infrastructure categories, and refer to these as negative examples.
The counts and proportions of the positive and negative classes in the dataset
are shown in Table 1.

###  2.3 Remotely sensed imagery

We pair all of the locations in the dataset with three publicly available
remotely sensed image sources. Specifically we include aerial imagery from the
USDA National Agriculture Imagery Program (NAIP) as well as satellite imagery
captured by Sentinel-1 (S1) and Sentinel-2 (S2). NAIP imagery covers the
contiguous U.S. and S1 and S2 imagery both have global coverage. For NAIP we
use 1m resolution imagery, for Sentinel-2 we use the L1C product at 10m
resolution, and for Sentinel-1 we use the Sigma Nought Backscatter product at
10m resolution. We use all spectral bands from each product. Specifically, we
use the three visible (RGB) and single near-infrared (NIR) bands from NAIP and
S2, the single coastal aerosol (CA) band, four red-edge (RE1-4) bands, single
water vapor (WV) band, single cirrus (C) band, and the two shortwave infrared
(SWIR1-2) bands from S2, and the V-transmit (VH and VV) bands from S1. We
include S1 and S2 in the dataset in order to enable experimenting with coarser
resolution satellite imagery which is globally available, unlike NAIP. The
details of each imagery product and band are shown in Table 2.

In order to construct images containing each location in the dataset, we
consider a 720m x 720m footprint centered around the location. This footprint
was chosen to balance the size of the images with the contextual information,
but we investigate this choice in the experiments. Due to the geographic
coordinate noise in the publicly available datasets, we chose to center the
imagery at the locations which increases the likelihood the facilities are
captured in the imagery, but still has natural variation in the locations of
the facilities in the imagery. We construct a mosaic of the most recently
captured pixels in a time range for each image product, where we consider NAIP
images captured between 2017 and 2021 and Sentinel-1 and Sentinel-2 images
between May and September 2021, where Sentinel-2 images are selected based on
lowest cloud cover. We use the Descartes Labs platform to download all of the
imagery [14].

The total dataset contains 86,599 images capturing ten spectral bands across
the three imagery products. Information about the remotely sensed image
products and bands included in the dataset are provided in Table 2 and
characteristic examples for each methane source category are shown in Figure 2
in the Appendix.

Table 2: Summary of the remotely sensed image products and bands included in
METER-ML. RGB are the three visible bands, NIR is a single near-infrared band,
RE1-4 are the four red-edge bands, SWIR1-2 are the two shortwave infrared
bands, CA is the single coastal aerosol band, WV is the single water-vapor
band, C is the single cirrus band, and VH & VV are the two V-transmit bands.

Product | Bands | Image Size | Resolution
---|---|---|---
NAIP | RGB & NIR | 720x720 | 1m
Sentinel-2 | RGB & NIR | 72x72 | 10m
Sentinel-2 | RE1-4 & SWIR1-2 | 36x36 | 20m
Sentinel-2 | CA & WV & C | 12x12 | 60m
Sentinel-1 | VH & VV | 72x72 | 10m

###  2.4 Validation and test sets

Two Stanford University postdoctoral researchers with expertise in methane
emissions and related infrastructure individually reviewed 1,533 examples to
compose the held-out validation and test sets. To determine which examples to
include in these held-out sets, we randomly sampled 150 images from each of
the six positive classes as well as a random sample of 33 images which have
multiple labels, constituting 933 positive examples according to the original
public dataset labels. We additionally sampled 12 images from each of the 50
negative categories resulting in 600 negative examples. The experts both
manually reviewed these examples and identified the presence or absence of the
six methane source categories by using a combination of NAIP imagery as well
as Google Maps imagery, which often had finer spatial resolution as well as
place names. The facility had to be captured by the NAIP image for the
corresponding label to be assigned. If the expert identified no clearly
visible methane source categories in the image, the example was labeled
“negative”, and if the expert was uncertain about any label, the example was
labeled “uncertain”. The two labels per example were then resolved as follows:

  1. 1.

If the experts agreed and neither was uncertain, the agreed upon label was
taken as the final label.

  2. 2.

If the experts disagreed, and one was uncertain but the other was not, the
expert’s certain label was taken as the final label.

  3. 3.

If the experts disagreed, but one agreed with the original label, the original
label was taken as the final label.

  4. 4.

In all other scenarios, the example was reviewed jointly by the experts and a
final label was assigned.

Only 76 examples out of the 1,533 went to another round of review. The
resulting datasets have 858 positive examples and 675 negative examples. We
split the 1,533 examples into 515 for the validation set and 1,018 for the
test set. The label counts on the validation and test sets are shown in Table
1.

##  3 Experiments

We run a variety of multi-label classification experiments on the curated
dataset. In all of our experiments, we use a DenseNet-121 convolutional neural
network architecture [15]. Preliminary experiments on the dataset explored
various ResNet and DenseNet architectures and found that DenseNets
outperformed all ResNet variants [16]. We use a linear layer which outputs six
values indicating the likelihood that each of the six methane source
categories are present in the input image, which outperformed individual
models across all classes in our preliminary experiments. Although the model
does not explicitly produce a value indicating the likelihood that the image
is negative, a low value assigned to all classes indicates a negative
prediction. The loss function is the mean of six unweighted binary cross
entropy losses, where the label is 1 if the class if present in the image and
0 otherwise. All six labels in the negative examples are 0. The network
weights are initialized with weights from a network pre-trained on ImageNet
[17]. Before inputting the images into the networks, we upscale the Sentinel-1
and Sentinel-2 images to match the size of NAIP images using bilinear
resampling and normalize the values by the display range of the bands (see
Table 7 in the Appendix). When using inputs with less than or more than 3
channels, we replace the first convolutional neural network layer with one
which accepts the corresponding number of channels. Each model is trained for
5 epochs with a batch size of 4. For each model we use the checkpoint saved
after an epoch which led to the lowest validation loss. We use an Adam
optimizer with standard parameters [18] and a learning rate of 0.02. All
models are trained using a GeForce GTX 1070 GPU.

The baseline setting for all experiments uses images capturing a footprint of
720m x 720m with 1m spatial resolution (720 x 720 image dimensions). After the
models are trained, each of the six values output by the model are fed through
an element-wise sigmoid function to produce a probability for each of the six
categories. To evaluate the performance of the models, we compute the per-
class area under the precision recall curve (AUPRC) and summarize the
performance over all classes by taking the macro-average of the per-class
AUPRCs.

Table 3: Per-class and overall (macro-average) validation AUPRC for different remotely sensed image products and bands. All of these experiments use images of size 720x720 at a spatial resolution of 1m per pixel, with S1 and S2 upsampled to that resolution. Image Product | Bands | CAFOs | Landfills | Mines | Proc Plants | R&Ts | WWTPs | Overall
---|---|---|---|---|---|---|---|---
S1 | VH&VV | 0.519 | 0.107 | 0.152 | 0.218 | 0.487 | 0.119 | 0.267
S2 | RGB | 0.889 | 0.268 | 0.305 | 0.374 | 0.694 | 0.204 | 0.456
S2 | All | 0.889 | 0.189 | 0.382 | 0.368 | 0.690 | 0.183 | 0.450
S2 & S1 | All | 0.923 | 0.152 | 0.379 | 0.391 | 0.612 | 0.231 | 0.448
NAIP | RGB | 0.903 | 0.270 | 0.348 | 0.327 | 0.849 | 0.182 | 0.480
NAIP | All | 0.945 | 0.276 | 0.401 | 0.508 | 0.857 | 0.303 | 0.548
NAIP & S2 & S1 | All | 0.889 | 0.214 | 0.473 | 0.457 | 0.796 | 0.272 | 0.517

###  3.1 Impact of using different imaging products and bands

We investigate the impact of using different combinations of image products
and bands in the dataset (Table 3). Specifically, we experiment with NAIP, S2,
and S1 alone, only visible bands and all spectral bands for S2 and NAIP, all
spectral bands from S1 and S2 together (representing the model closest to
public global transferability due to the global coverage of S1 and S2), and
all spectral bands from the three products together.

The best model according to macro-average AUPRC is the one which uses NAIP
with all bands (the three visible bands and NIR band), achieving an overall
AUPRC of 0.548 and the highest performance on CAFOs, Landfills, Proc Plants,
R&Ts, and WWTPs compared to all other tested product and band combinations.
Notably, it achieves very high performance on CAFOs (AUPRC=0.945) and high
performance on R&Ts (AUPRC=0.857). The second best model is the joint
NAIP+S2+S1 model, achieving an overall AUPRC of 0.517 and the highest
performance on Mines (AUPRC=0.473) compared to all other tested product and
band combinations.

S1 alone underperforms all other combinations of products and bands, followed
by S2 and S1 jointly, which performed similarly overall to S2 with only the
visible bands and all spectral bands. Importantly, the S2 and S1 joint model
still achieves high performance on CAFOs (AUPRC=0.923), although the
performance is lower than performance on CAFOs using NAIP imagery
(AUPRC=0.947). There is a significant drop in performance on all classes when
moving from NAIP to S2, highlighting the benefit of using high spatial
resolution imagery.

The inclusion of the non-visible information substantially improves overall
AUPRC for NAIP (AUPRC=0.480 $\rightarrow$ 0.548) but not for Sentinel 2
(AUPRC=0.450 $\rightarrow$ 0.448). For NAIP, the improvement is observed for
all classes, with substantial gains on CAFOs, Mines, Proc Plants, and WWTPs.
For Sentinel 2, the inclusion of non-visible bands substantially improves
performance on Mines but substantially degrades performance on Landfills. For
both products, minimal change on R&Ts performance is observed when including
the non-visible bands.

Table 4: Per-class and overall (macro-average) validation AUPRC at varying image footprints and spatial resolutions. Image Footprint | Resolution | CAFOs | Landfills | Mines | Proc Plants | R&Ts | WWTPs | Overall
---|---|---|---|---|---|---|---|---
240x240 | 1m | 0.773 | 0.217 | 0.407 | 0.438 | 0.735 | 0.337 | 0.485
480x480 | 1m | 0.772 | 0.226 | 0.260 | 0.371 | 0.855 | 0.506 | 0.498
720x720 | 3m | 0.891 | 0.245 | 0.378 | 0.566 | 0.837 | 0.269 | 0.531
720x720 | 1.5m | 0.927 | 0.244 | 0.426 | 0.366 | 0.831 | 0.449 | 0.541
720x720 | 1m | 0.945 | 0.276 | 0.401 | 0.508 | 0.857 | 0.303 | 0.548
Table 5: Per-class and overall (macros-average) test metrics of the per-class expert model. The per-class expert model consists of one model per class, where the model used for each class is selected based on the highest performing settings for that class across the product, bands, footprint, and resolution experiments. Category | AUPRC | AUROCC | Precision | Recall | F1
---|---|---|---|---|---
CAFOs | 0.915 | 0.989 | 0.822 | 0.902 | 0.860
Landfills | 0.259 | 0.754 | 0.246 | 0.523 | 0.334
Mines | 0.470 | 0.905 | 0.558 | 0.403 | 0.468
Proc Plants | 0.350 | 0.787 | 0.336 | 0.477 | 0.394
R&Ts | 0.821 | 0.956 | 0.752 | 0.787 | 0.769
WWTPs | 0.534 | 0.836 | 0.633 | 0.477 | 0.544
Overall | 0.558 | 0.871 | 0.558 | 0.595 | 0.562

###  3.2 Impact of image footprint and spatial resolution

As image footprint (i.e. the amount of area on the ground captured by the
image) and spatial resolution likely impact model performance due to the
variation in the sizes of the methane-emitting facilities and equipment, we
conduct experiments to test these effects (Table 4). To investigate the impact
of footprint, we center crop the 720 x 720 1m images to obtain 480 x 480 and
240 x 240 1m images corresponding to 480m x 480m and 240m x 240m footprints
respectively. Note that this reduces the area on the ground with spatial
resolution held constant. To investigate the impact of spatial resolution, we
use cubic spline interpolation [19] to downsample the 720 x 720 images to 480
x 480 (1.5m resolution, corresponding to Airbus SPOT imagery) and 240 x 240
(3m resolution, corresponding to PlanetScope imagery). Note that this reduces
the spatial resolution without modifying the image footprint. In all
experiments, we up-sample the images back to 720 x 720 to avoid any
differences in performance due to varying image size. We use NAIP with RGB +
NIR bands for these experiments as this setting produced the best overall
performance compared to the other combinations of products and bands.

We find that the largest tested image footprint achieves the highest overall
performance (0.548) and substantially outperforms both smaller spatial
footprints across all classes except for WWTPs. This may be explained by the
fact that a significant number of smaller wastewater treatment plants are
surrounded by industrial buildings and other infrastructure, so cropping out
this infrastructure improves the model’s ability to identify the salient
features of the wastewater treatment facilities.

We further find that the highest spatial resolution achieves the best overall
performance (AUPRC=0.548), outperforming the coarser resolution models on
CAFOs, Landfills, and R&Ts. The 1.5m resolution model closely follows with an
overall AUPRC of 0.541 and outperforms the 1m resolution model on Mines. The
3m resolution model also closely follows the 1.5m resolution model achieving
an overall performance of 0.531, and substantially outperforms both the higher
resolution models on Proc Plants. This result suggests that models developed
at 1.5m and even 3m resolution have the potential to perform almost as well as
1m resolution models, which has implications on global applicability as Airbus
SPOT and PlanetScope are globally (privately) available at 1.5m and 3m
resolution respectively.

###  3.3 Per-class expert model test set results

For each methane source category, we select the experimental configuration
(product/band/footprint/resolution) that achieved the highest validation AUPRC
for that class to serve as the “class expert”. We refer to the combination of
the different class experts as the per-class expert model.

We evaluate the per-class expert model on the hold-out test set using a
variety of metrics including AUPRC and area under the receiver operating
characteristic curve (AUROCC) as well as precision, recall, and F1 at the
threshold which achieves the highest F1 on the validation set. The results are
shown in Table 5. The per-class expert model obtains a macro-average AUPRC of
0.558. The model does especially well on CAFOs (AUPRC=0.915) and R&Ts
(AUPRC=0.821), possibly because these sources have very distinctive features
(e.g., long barns in CAFOs and storage tank farms in R&Ts). It performs more
poorly on the other sources, especially landfills which do not have many clear
distinctive features visible at 1m resolution. Notably it achieves the lowest
performance on the categories with the least number of examples in the
dataset, excluding R&Ts which may be simpler to identify due to their
homogeneity and discernible features.

##  4 Discussion

The experiments suggest that the choice of imaging product, spectral band,
image footprint, and spatial resolution can lead to substantial differences in
model performance, with the effect often depending on the methane source
category. In particular, this suggests that there is significant room to
explore approaches which leverage the multi-sensor and multi-spectral aspects
of METER-ML. For example, the NAIP & S2 & S1 model underperformed the model
which used NAIP alone, and using all 13 spectral bands in the S2 model did not
lead to substantial performance differences compared to the S2 model which
only used the three visible bands. We also do not leverage the geographic
information explicitly in the models, but this has been shown to improve
performance on other Earth observation tasks [20, 21]. Furthermore, there is
potential to augment the dataset with other sources of imagery and information
available at the provided geographic locations. We hope to help create new
versions of METER-ML which may include other sources of input data and methane
emitting infrastructure categories.

The best model from our experiments achieves high performance on identifying
CAFOs and R&Ts, suggesting the potential to map these facilities with NAIP
imagery in the U.S. which aligns with findings from prior studies [9, 10]. The
performance for identifying CAFOs remains high when using S1 and S2, which are
globally and publicly available. This suggests the potential to use these
lower spatial resolution imagery sources to map CAFOs in other countries
besides the U.S., but future work should investigate whether these findings
generalize to other regions. There is still a large gap to achieving high
performance for each of the other methane source categories and further
improve performance on the high performing categories, so METER-ML is a
challenging benchmark to test new infrastructure identification approaches.

There are many other publicly available remote sensing datasets for
classification, with some of the most common being UC Merced [22], SAT-4 and
SAT-6 [23], AID [24], NWPU-RESISC45 [25], EuroSAT [26], and BigEarthNet [27].
Few of these datasets have georeferenced multi-sensor images, which limits
their utility for new modeling approaches and downstream use. The OGNet
dataset [9] is the most similar publicly available dataset to METER-ML and is
essentially a subset of it, containing NAIP imagery of refineries in the
contiguous U.S.

We identify four limitations of this work. First, we limit the geographic
scope of METER-ML to the U.S. due to the availability of disseminatable
infrastructure data and publicly available, high resolution imagery. Future
work should include data in other regions worldwide. Second, we do not include
longitudinal imagery in the dataset to reduce the size and complexity of the
dataset as most infrastructure is static over time. However, longitudinal
information has the potential to provide additional signal to help
differentiate certain facilities, e.g. waste pile evolution at landfills.
Third, we use a DenseNet121 model that is pre-trained on ImageNet, but the
shape and number of channels of remote sensing imagery can be significantly
different from ImageNet. It would be worthwhile to train a network from
scratch on METER-ML, and compare its performance against a network that is
pre-trained on ImageNet and fine-tuned on METER-ML. Fourth, our approach to
combine the multi-sensor data may not be optimal as the products and spectral
bands have different spatial resolutions and sensor types (e.g. active vs.
passive sensors). One alternate approach may be to dedicate different network
branches for the inputs and combine the representations from each branch.

##  5 Conclusion

In this work, we curate a large georeferenced multi-sensor dataset called
METER-ML to test automated methane source identification approaches. We
conduct a variety of experiments investigating the impact of remotely sensed
image product, spectral bands, image footprint, and spatial resolution on
model performance measured against a consensus of expert labels. We find that
a model which leverages NAIP with all four bands achieves the highest overall
performance across the tested image product and spectral band combinations,
followed closely by a joint NAIP, Sentinel-2, and Sentinel-1 model. We also
find that the highest spatial resolution and footprint leads to the best
overall performance, although performance can depend on the methane source
category. Finally we show that the best model achieves high performance in
identifying concentrated animal feeding operations and oil refineries and
petroleum terminals, suggesting the potential to map them at scale, but
substantially lower performance on the other four categories with notably
lower performance identifying processing plants and landfills. We make METER-
ML freely available in order to encourage and support future work on
developing Earth observation models for mitigating climate change.

###### Acknowledgements.

This work was supported by the High Tide Foundation to construct the METER
database. We acknowledge Rose Rustowicz and Kyle Story for their support of
this work, as well as the Descartes Labs Platform API and tools for
downloading and processing the remotely sensed imagery. We also thank Ritesh
Gautam and Mark Omara for their help working with the oil and gas
infrastructure data, Evan Sherwin for his advice on the dataset and methane
source categories, and Victor Maus for providing the coal mines data.

## References

  * Zhang et al. [2022] Z. Zhang, B. Poulter, S. Knox, A. Stavert, G. McNicol, E. Fluet-Chouinard, A. Feinberg, Y. Zhao, P. Bousquet, J. G. Canadell, et al.,  Anthropogenic emission is the main contributor to the rise of atmospheric methane during 1993–2017,  National science review 9 (2022) nwab200.
  * Agreement [2015] P. Agreement,  Paris agreement,  in: Report of the Conference of the Parties to the United Nations Framework Convention on Climate Change (21st Session, 2015: Paris). Retrived December, volume 4, HeinOnline, 2015, p. 2017\.
  * Jacob et al. [2022] D. J. Jacob, D. J. Varon, D. H. Cusworth, P. E. Dennison, C. Frankenberg, R. Gautam, L. Guanter, J. Kelley, J. McKeever, L. E. Ott, et al.,  Quantifying methane emissions from the global scale down to point sources using satellite observations of atmospheric methane,  Atmospheric Chemistry and Physics Discussions (2022) 1–44.
  * Yu et al. [2018] J. Yu, Z. Wang, A. Majumdar, R. Rajagopal,  Deepsolar: A machine learning framework to efficiently construct a solar deployment database in the united states,  Joule 2 (2018) 2605–2617.
  * Lee et al. [2021] J. Lee, N. R. Brooks, F. Tajwar, M. Burke, S. Ermon, D. B. Lobell, D. Biswas, S. P. Luby,  Scalable deep learning to identify brick kilns and aid regulatory capacity,  Proceedings of the National Academy of Sciences 118 (2021).
  * Kruitwagen et al. [2021] L. Kruitwagen, K. Story, J. Friedrich, L. Byers, S. Skillman, C. Hepburn,  A global inventory of photovoltaic solar energy generating units,  Nature 598 (2021) 604–610.
  * Sirko et al. [2021] W. Sirko, S. Kashubin, M. Ritter, A. Annkah, Y. S. E. Bouchareb, Y. Dauphin, D. Keysers, M. Neumann, M. Cisse, J. Quinn,  Continental-scale building detection from high resolution satellite imagery,  arXiv preprint arXiv:2107.12283 (2021).
  * Dileep et al. [2020] S. Dileep, D. Zimmerle, J. R. Beveridge, T. Vaughn,  Automated identification of oil field features using cnns,  in: NeurIPS, Workshop on Tackling Climate Change with Machine Learning, 2020, 2020.
  * Sheng et al. [2020] H. Sheng, J. Irvin, S. Munukutla, S. Zhang, C. Cross, K. Story, R. Rustowicz, C. Elsworth, Z. Yang, M. Omara, et al.,  Ognet: Towards a global oil and gas infrastructure database using deep learning on remotely sensed imagery,  arXiv preprint arXiv:2011.07227 (2020).
  * Handan-Nader and Ho [2019] C. Handan-Nader, D. E. Ho,  Deep learning to map concentrated animal feeding operations,  Nature Sustainability 2 (2019) 298–306.
  * Li et al. [2022] H. Li, J. Zech, D. Hong, P. Ghamisi, M. Schultz, A. Zipf,  Leveraging openstreetmap and multimodal remote sensing data with joint deep learning for wastewater treatment plants detection,  International Journal of Applied Earth Observation and Geoinformation 110 (2022) 102804\.
  * Stanford University [2022] Stanford University, Methane tracking emissions reference platform, 2022. URL: https://meterplatform.web.app/, accessed: 2022-06-30.
  * Keisler et al. [2019] R. Keisler, S. W. Skillman, S. Gonnabathula, J. Poehnelt, X. Rudelis, M. S. Warren,  Visual search over billions of aerial and satellite images,  Computer Vision and Image Understanding 187 (2019) 102790.
  * Labs [2022] D. Labs, Descartes labs platform, 2022\. URL: https://descarteslabs.com/, accessed: 2022-06-26.
  * Huang et al. [2017] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger,  Densely connected convolutional networks,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708.
  * He et al. [2016] K. He, X. Zhang, S. Ren, J. Sun,  Deep residual learning for image recognition,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei,  Imagenet: A large-scale hierarchical image database,  in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.
  * Kingma and Ba [2014] D. P. Kingma, J. Ba,  Adam: A method for stochastic optimization,  arXiv preprint arXiv:1412.6980 (2014).
  * Parsania and Virparia [2016] P. S. Parsania, P. V. Virparia,  A comparative analysis of image interpolation algorithms,  International Journal of Advanced Research in Computer and Communication Engineering 5 (2016) 29–34.
  * Mac Aodha et al. [2019] O. Mac Aodha, E. Cole, P. Perona,  Presence-only geographical priors for fine-grained image classification,  in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 9596–9606.
  * Irvin et al. [2020] J. Irvin, H. Sheng, N. Ramachandran, S. Johnson-Yu, S. Zhou, K. Story, R. Rustowicz, C. Elsworth, K. Austin, A. Y. Ng,  Forestnet: Classifying drivers of deforestation in indonesia using deep learning on satellite imagery,  arXiv preprint arXiv:2011.05479 (2020).
  * Yang and Newsam [2012] Y. Yang, S. Newsam,  Geographic image retrieval using local invariant features,  IEEE Transactions on Geoscience and Remote Sensing 51 (2012) 818–832.
  * Basu et al. [2015] S. Basu, S. Ganguly, S. Mukhopadhyay, R. DiBiano, M. Karki, R. Nemani,  Deepsat: a learning framework for satellite imagery,  in: Proceedings of the 23rd SIGSPATIAL international conference on advances in geographic information systems, 2015, pp. 1–10.
  * Xia et al. [2017] G.-S. Xia, J. Hu, F. Hu, B. Shi, X. Bai, Y. Zhong, L. Zhang, X. Lu,  Aid: A benchmark data set for performance evaluation of aerial scene classification,  IEEE Transactions on Geoscience and Remote Sensing 55 (2017) 3965–3981.
  * Cheng et al. [2017] G. Cheng, J. Han, X. Lu,  Remote sensing image scene classification: Benchmark and state of the art,  Proceedings of the IEEE 105 (2017) 1865–1883.
  * Helber et al. [2019] P. Helber, B. Bischke, A. Dengel, D. Borth,  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12 (2019) 2217–2226.
  * Sumbul et al. [2019] G. Sumbul, M. Charfuelan, B. Demir, V. Markl,  Bigearthnet: A large-scale benchmark archive for remote sensing image understanding,  in: IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, IEEE, 2019, pp. 5901–5904.
  * CA Governor’s Office of Emergency Services [2021] CA Governor’s Office of Emergency Services, Ca energy commission - oil refineries and terminals, 2021\. URL: https://arcg.is/1rXn4G0, accessed: 2022-02-01.
  * CA State Water Resources Control Board [2022] CA State Water Resources Control Board, 2022\. URL: https://www.waterboards.ca.gov/resources/data_databases/site_map.html, accessed: 2022-02-01.
  * McKernan [2012] D. McKernan, 2012 factory farms in michigan, 2012\. URL: https://data.world/dmckernan/2012-factory-farms-in-michigan, accessed: 2022-02-01.
  * Tsubiks [2017] O. Tsubiks, Concentrated animal feeding operations (cafo), us, 2017. URL: https://data.world/dataforacause/concentrated-animal-feeding-operations-cafo, accessed: 2022-02-01.
  * U.S. Energy Information Administration [2021] U.S. Energy Information Administration, U.s. energy mapping system, 2021. URL: https://www.eia.gov/state/maps.php, accessed: 2022-02-01.
  * U.S. Environmental Protection Agency [2021] U.S. Environmental Protection Agency, Greenhouse gas reporting program (ghgrp), 2021\. URL: https://www.epa.gov/ghgreporting, accessed: 2022-02-01.
  * Rose et al. [2018] K. Rose, J. Bauer, V. Baker, A. Barkhurst, A. Bean, J. DiGiulio, K. Jones, T. Jones, D. Justman, R. Miller, et al., Global Oil & Gas Features Database, Technical Report, 2018.
  * U.S. D of Homeland Security [2021] U.S. D of Homeland Security, Homeland infrastructure foundation-level data (hifld), 2021\. URL: https://hifld-geoplatform.opendata.arcgis.com/, accessed: 2022-02-01.
  * Ehalt Macedo et al. [2022] H. Ehalt Macedo, B. Lehner, J. Nicell, G. Grill, J. Li, A. Limtong, R. Shakya,  Distribution and characteristics of wastewater treatment plants within the global river network,  Earth System Science Data 14 (2022) 559–577.
  * Indiana D of Environmental Management [2020] Indiana D of Environmental Management, Confined feeding operation facilities: Indiana, 2020\. URL: https://maps.indiana.edu/metadata/Environment/Agribusiness_Confined_Feeding_Operations.html#Identification_Information, accessed: 2022-02-01.
  * U.S. Environmental Protection Agency [2022] U.S. Environmental Protection Agency, Landfill methane outreach program, 2022. URL: https://www.epa.gov/lmop/project-and-landfill-data-state, accessed: 2022-02-01.
  * Marchese et al. [2015] A. J. Marchese, T. L. Vaughn, D. J. Zimmerle, D. M. Martinez, L. L. Williams, A. L. Robinson, A. L. Mitchell, R. Subramanian, D. S. Tkacik, J. R. Roscioli, et al.,  Methane emissions from united states natural gas gathering and processing,  Environmental science & technology 49 (2015) 10718–10727.
  * Maus et al. [2022] V. Maus, S. Giljum, D. M. da Silva, J. Gutschlhofer, R. P. da Rosa, S. Luckeneder, S. L. Gass, M. Lieber, I. McCallum,  An update on global mining land use,  Scientific data (2022).
  * Metropolitan Council [2020] Metropolitan Council, Wastewater treatment plants, 2020. URL: https://gisdata.mn.gov/dataset/us-mn-state-metc-util-wastewater-treatment-plants, accessed: 2022-02-01.
  * MN Pollution Control Agency [2016] MN Pollution Control Agency, Feedlots in minnesota, 2016. URL: https://gisdata.mn.gov/dataset/env-feedlots, accessed: 2022-02-01.
  * ORNL DAAC [2019] ORNL DAAC, Sources of methane emissions (vista-ca), state of california, usa, 2019. URL: https://daac.ornl.gov/NACP/guides/NACP_Vista_CA_CH4_Inventory.html, accessed: 2022-02-01.
  * Sierra Club Michigan Chapter [2017] Sierra Club Michigan Chapter, Michigan cafo mapping report, 2017. URL: https://www.sierraclub.org/michigan/michigan-cafo-mapping-report, accessed: 2022-02-01.
  * Stanford Regulation, Evaluation, and Governance Lab [2019] Stanford Regulation, Evaluation, and Governance Lab, Cafo training dataset, 2019. URL: https://reglab.stanford.edu/data/cafo-training-dataset/, accessed: 2022-02-01.

##  Appendix A Appendix

###  A.1 Methane-Emitting Infrastructure Datasets

Table 6: Summary of the datasets containing the locations of methane-emitting
infrastructure that are included in METER-ML. All datasets are subsetted to
the locations within the contiguous U.S. We use the centroids of polygons for
any datasets provided in polygon format.

Dataset Source | Scope | Methane Source Categories
---|---|---
CA Energy Commission [28] | California | R&Ts
CSWRCB [29] | California | WWTPs
data.world [30] | Michigan | CAFOs
Data For Cause Challenge [31] | US | CAFOs
EIA [32] | US | Proc Plants, R&Ts, WWTPs
GHGRP [33] | US | Landfills, Mines, R&Ts, WWTPs
GOGI [34] | Global | Proc Plants, R&Ts
HIFLD [35] | US | R&Ts, WWTPs
HydroWASTE [36] | Global | WWTPs
IndianaMap [37] | Indiana | CAFOs
LMOP [38] | US | Landfills
Marchese et al. [39] | US | Proc Plants
Maus et al. [40] | Global | Mines
Minnesota Metropolitan Council [41] | Minnesota | WWTPs
Minnesota Pollution Control Agency [42] | Minnesota | CAFOs
ORNL DAAC [43] | California | CAFOs
Sierra Club [44] | Michigan | CAFOs
Stanford RegLab [45] | North Carolina | CAFOs

####  A.1.1 Coal Mines Data

The mines data from [40] were subsetted to coal mines in order to capture the
mines responsible for the vast majority of methane emissions. To do this, the
polygons and coal mine coordinates obtained from S&P Global Commodity Insights
were matched to determine which polygons were spatially related to coal mine
coordinates. Then a visual check and hand cleaning was performed on the
polygons assigned a coal mining label to ensure correctness.

###  A.2 Negative Classes

We identify a variety of infrastructure which are not any of the six
infrastructure categories to use as negatives in the dataset. Specifically we
include football fields, marinas, solar panels, large bodies of water, parking
lots, windmills, baseball fields, airport runways, clouds, neighborhoods, golf
courses, roundabouts, mountainous terrain, trees, boats, islands, rocks,
rivers, roads, bridges, ripples in water, snow, canyon formations, sparse
forests, suburban neighborhoods, beaches, clear water, swimming pools, sand,
corn farms, soy farms, trees on mountainside, farm houses, grass, airplanes,
turning roads, intersections, multifamily residential facilities, rapids,
docks, highway loops, mowed grass, container yards, soccer fields,
greenhouses, crops, personal watercrafts, pivot irrigation systems, and
concrete plants. Characteristic examples of each type were selected and a
variety of similar examples per type were obtained using the Descartes Labs
GeoVisual Similarity tool [13].

###  A.3 Remotely Sensed Image Statistics and Examples

Table 7: Summary of the remotely sensed image products and bands included in
METER-ML. The raw image data contains values from the data range, while the
display range is used to normalize values before displaying imagery or
inputting into models.

Product/Bands | Image Size | Resolution | Data Range | Display Range
---|---|---|---|---
NAIP RGB & NIR | 720x720 | 1m | [0,255] | [0,255]
Sentinel-2 CA | 12x12 | 60m | [0,10000] | [0,10000]
Sentinel-2 RGB | 72x72 | 10m | [0,10000] | [0,4000]
Sentinel-2 RE1-4 | 36x36 | 20m | [0,10000] | [0,10000]
Sentinel-2 NIR | 72x72 | 10m | [0,10000] | [0,10000]
Sentinel-2 WV | 12x12 | 60m | [0,10000] | [0,10000]
Sentinel-2 C | 12x12 | 60m | [0,10000] | [0,10000]
Sentinel-2 SWIR1-2 | 36x36 | 20m | [0,10000] | [0,10000]
Sentinel-1 VH | 72x72 | 10m | [1,4095] | [585,2100]
Sentinel-1 VV | 72x72 | 10m | [1,4095] | [585,2926]

Figure 2: Characteristic example images of each category in METER-ML. METER-ML contains 19 spectral bands across 3 image products, visualized here in grayscale for single bands and false color composites for multiple bands (the first listed band is used in the red channel, second in the green, third in the blue). Category | NAIP RGB | NAIP NIR | S1 VV&VH
---|---|---|---
CAFOs |  |  |
Coal Mines |  |  |
Landfills |  |  |
Proc Plants |  |  |
R&Ts |  |  |
WWTPs |  |  |
Negatives |  |  |
Category | S2 RGB | S2 NIR | S2 RE1&SWIR1-2 | S2 RE2-4 | S2 CA&WV&C
---|---|---|---|---|---
CAFOs |  |  |  |  |
Coal Mines |  |  |  |  |
Landfills |  |  |  |  |
Proc Plants |  |  |  |  |
R&Ts |  |  |  |  |
WWTPs |  |  |  |  |
Negatives |  |  |  |  |


# Fast Clustering of Short Text Streams Using Efficient Cluster Indexing and
Dynamic Similarity Thresholds

Md Rashadul Hasan Rakib  Dalhousie UniversityNova ScotiaCanada<EMAIL_ADDRESS>and  Muhammad Asaduzzaman  Queen’s UniversityOntarioCanada
<EMAIL_ADDRESS>
(2020)

###### Abstract.

Short text stream clustering is an important but challenging task since
massive amount of text is generated from different sources such as micro-
blogging, question-answering, and social news aggregation websites. One of the
major challenges of clustering such massive amount of text is to cluster them
within a reasonable amount of time. The existing state-of-the-art short text
stream clustering methods can not cluster such massive amount of text within a
reasonable amount of time as they compute similarities between a text and all
the existing clusters to assign that text to a cluster. To overcome this
challenge, we propose a fast short text stream clustering method (called
FastStream) that efficiently index the clusters using inverted index and
compute similarity between a text and a selected number of clusters while
assigning a text to a cluster. In this way, we not only reduce the running
time of our proposed method but also reduce the running time of several state-
of-the-art short text stream clustering methods. FastStream assigns a text to
a cluster (new or existing) using the dynamically computed similarity
thresholds based on statistical measure. Thus our method efficiently deals
with the concept drift problem. Experimental results demonstrate that
FastStream outperforms the state-of-the-art short text stream clustering
methods by a significant margin on several short text datasets. In addition,
the running time of FastStream is several orders of magnitude faster than that
of the state-of-the-art methods.

text stream clustering, inverted index, dynamic similarity threshold

††copyright: acmcopyright††journalyear: 2020††conference: aa ’20: aa; 2020;
USA††booktitle: aaa ’20: aa, June 03–05, 2020, USA††price: 15.00††journalyear:
2020††copyright: acmcopyright††conference: aa 2020; September 29-October 2,
2020; Virtual Event, CA, USA††booktitle: aa 2020 (aa ’20), September
29-October 2, 2020, Virtual Event, CA, USA††price: 15.00††doi:
10.1145/3395027.3419589††isbn: 978-1-4503-8000-3/20/09††ccs: Computing
methodologies Natural language processing††ccs: Computing methodologies
Cluster analysis

##  1\. Introduction

Due to technological advances short text streams are generated at large
volumes from different sources. Organizing these text streams within
reasonable time-frame is an important step towards discovering real-time
trends (e.g., political, economic) in conversations, finding groups of users
sharing similar topics of interest (Zhao et al., 2016), and monitoring the
activity of individuals over time. The task of clustering short text streams
is to assign a text to a new cluster or to one of the existing clusters as it
arrives (Yin et al., 2018). The recent short text stream clustering methods
were proposed based on dirichlet process multinational mixture model as
described in (Yin et al., 2018; Kumar et al., 2020; Chen et al., 2020).
Generally, these algorithms used Gibbs Sampling (Ishwaran and James, 2001) to
estimate the parameters of the mixture model so as to obtain the clustering of
text streams. These algorithms assign a text to a cluster by computing
similarities between a text and the clusters based on the common features
between them (e.g., word, biterm). However, there may be some clusters with
which a text may not share any common feature, thus the similarities between
that text and those clusters are zero. Therefore, those similarity
computations can be ignored.

Motivated by this, FastStream selects a specific set of clusters while
computing similarities between a text and the clusters. These selected
clusters share common features with a particular text. By limiting the number
of similarity computations, we significantly reduce the running time of our
proposed method and thus the running time of our method is several orders of
magnitude faster than that of the state-of-the-art methods.

In general, similarity-based text stream clustering methods use user defined
similarity threshold to assign a text to a new or one of the existing clusters
(Yin et al., 2018). Rakib et al. (2020) proposed a short text stream
clustering method that removes outliers from the clusters and reassigns them
to the clusters using dynamically computed similarity thresholds (Rakib et
al., 2020). This motivates us to dynamically calculate the similarity
threshold for each text based on statistical measure and use this similarity
threshold to assign the text to a new or one of the existing clusters. Thus
our method efficiently handles the _concept drift_ problem (Yin et al., 2018)
(the problem that topics of the text streams may change over time).

The three major contributions of our work are as follows.

  * •

First, we improve the running time of our
method111https://github.com/rashadulrakib/short-text-stream-
clustering/tree/master/OnlineClustering by computing the similarity between a
text and a selected number of clusters (obtained by inverted index (Ilic et
al., 2014)) instead of all clusters. Therefore, the running time of our method
is several orders of magnitude faster than that of the state-of-the-art short
text stream clustering methods. Moreover, the experimental results demonstrate
that by applying inverted index, we are able to reduce the running time of
several state-of-the-art methods.

  * •

Second, our method utilizes dynamically calculated similarity thresholds based
on the statistical measure to assign a text to a new or an existing cluster.
The use of dynamic thresholds helps to avoid the concept drift problem (Yin et
al., 2018).

  * •

Third, we perform an extensive comparison of our proposed method using four
different datasets and our method outperforms the state-of-the-art short text
stream clustering methods by a significant margin on several datasets.

##  2\. Related Work

###  2.1. Similarity-based Stream Clustering

In general, similarity-based text stream clustering methods use the vector
space model (Erk, 2012) to represent the documents. A document is assigned to
a new or one of the existing clusters based on the similarity threshold which
needs to be manually determined by the user (Yin et al., 2018). A detailed
survey of clustering data streams can be found in (Carnein and Trautmann,
2019). In our method, we dynamically calculate the similarity threshold for
each text and use this similarity threshold to assign the text to a new or one
of the existing clusters.

CluStream (Aggarwal et al., 2003) is a stream clustering method consisting of
an online micro-clustering phase and an offline macro-clustering phase. In
online phase, it assigns a data point to a new or an existing micro-cluster
based on a similarity threshold. In offline phase, it applies k-means
clustering on the micro-clusters and obtain k user specified macro-clusters.

Sumblr (Shou et al., 2013) is a tweet stream summarization prototype. It
consists of a text stream clustering module that compresses tweets into tweet
feature vectors (TCVs) and assigns future tweets to the clusters based on the
statistics of TCVs.

An efficient text stream clustering method using term burst information was
proposed in (Kalogeratos et al., 2016). Bursty terms are the terms that appear
in many documents during a short period of time. This approach considers the
fact that the documents that are published on a particular topic within a
certain time period contain a particular set of bursty terms. An user-defined
threshold for the number of occurrences of terms in documents is used to
identify bursty terms.

###  2.2. Model-based Stream Clustering

A recent short text stream clustering algorithm based on dirichlet process
multinational mixture model was proposed in (Yin et al., 2018) which uses two
dirichlet priors $\alpha$ and $\beta$. $\alpha$ refers to the prior
probability of a text choosing a new cluster and $\beta$ corresponds to the
prior probability of a text choosing a cluster with which the text shares more
similar content than other clusters. This algorithm has two variants: one is
by retaining all previous clusters (called MStream) and other one is by
removing old clusters (called MStreamF).

Rakib et al. (2020) proposed a short text stream clustering method that
clusters a fraction of texts in each batch2221A Batch (or stream) is defined
as a collection of texts (Yin et al., 2018). The terms “Batch” and “Stream”
are interchangeably used in our paper using the frequent biterms in texts,
then it populates the clustering model of MStream algorithm using the cluster
assignments obtained by the frequently occurred biterms in texts. After that
it clusters rest of the texts in the batch using the populated clustering
model of MStream algorithm. Following that it removes outliers from the
clusters and reassigns them to the clusters (new or existing) using the
dynamically computed similarity thresholds.

A biterm based mixture model for short text stream clustering was proposed in
(Chen et al., 2020). Similar to MStream(F) algorithm (Yin et al., 2018), the
biterm based clustering method developed two variants: one is by retaining the
clusters obtained in previous batches (called DP-BMM) and other is by
discarding the clusters obtained in previous batches (called DP-BMM-FP). The
main difference between MStream(F) and DP-BMM(-FP) is that DP-BMM(-FP)
represents the texts using biterm features instead of unigrams. In particular,
DP-BMM(-FP) represents a text of $n$ words using $n*(n-1)/2$ biterm features.

OSDM (Kumar et al., 2020) is a semantic-enhanced dirichlet model for short
text stream clustering. OSDM extends the MStream (Yin et al., 2018) algorithm
by integrating the word to word co-occurrence based semantic information
obtained from the common words between a text and a cluster and uses this
semantic information to compute similarity between a text and a cluster.

In general, a Dirichlet process mixture model based clustering algorithm
requires tuning the parameters (i.e., $\alpha$ and $\beta$) to obtain the
desired clustering performance. For example, MStream(F) uses $\alpha=0.03$ and
$\beta=0.03$ to obtain optimal clustering performance. The DP-BMM(-FP) and
OSDM performed grid search to obtain the optimal values for $\alpha$ and
$\beta$ and set ($\alpha=0.6$ and $\beta=0.02$) and ($\alpha=2e^{-3}$ and
$\beta=4e^{-5}$) respectively. On the contrary, our proposed method
(FastStream) does not require this kind of parameter tuning, instead it uses
the dynamically computed similarity threshold (based on statistical measure)
to assign a text to a new or an existing cluster.

DCT-L (Liang et al., 2016) is a dynamic clustering topic model for short text
streams based on dirichlet process multinomial mixture model. It assigns a
single topic (i.e., cluster) to each short text at a particular timestamp and
uses the resulting topic distribution as priors for inferring the topics of
subsequent documents.

DTM (Blei and Lafferty, 2006) is a dynamic topic model that analyzes the
topics of a collection of documents over time. This method assumes that a
document is rich enough to contain multiple topics. However, this assumption
does not work well for short texts, which results low quality performance on
short text streams.

##  3\. Background

###  3.1. Text Representation

We represent each text using three different kinds of features ($f$) which are
unigram, bigram and biterm and use each representation separately to cluster
the streams of texts. The words of a text are considered as the unigram
features of that text. The two consecutive words of a text are considered as
the bigram features of that text. The biterm feature is defined as an
unordered _word pair_ constructed using the words contained in a text (Chen et
al., 2020). For example, the text “ai improves healthcare system” will be
represented by the following bigrams: “ai improves”, “improves healthcare”,
and “healthcare system”. The same text will be represented by the following
biterms: “ai improves”, “ai healthcare”, “ai system”, “improves healthcare”,
“improves system”, and “healthcare system”.

###  3.2. Cluster Representation

In our method, each cluster is represented by a cluster feature ($CF$) vector
(Yin et al., 2018) consisting of 4 tuples {$n_{z}^{f}$, $n_{z}$, $m_{z}$,
$id_{z}$ } where $n_{z}^{f}$ refers to the features (unigram, bigram or
biterm) along with frequencies in cluster $z$, $n_{z}$ refers to the number of
features in cluster $z$, $m_{z}$ refers to the number of texts in cluster $z$,
and $id_{z}$ refers to the unique _id_ for cluster $z$.

###  3.3. Inverted Index

Inverted Index is a hashmap like data structure that creates mapping from
document-features (unigram, bigram or biterm) to documents (Ilic et al.,
2014). To keep track of which clusters are associated with which features, we
adopt the inverted index based searching technique and create a Feature vector
$F$ for each feature, defined as a tuple of {$l_{f}^{id}$} where $l_{f}^{id}$
refers to the list of cluster $ids$ associated with a feature $f$.

##  4\. Proposed Method

The proposed method (FastStream) clusters each short text one by one as it
arrives. At first, the features are extracted from the text. At a time, we use
only one type of feature to cluster the streams of texts. That means, we
extract only either unigram, bigram or biterm features from the text. After
that we select the clusters that contain the features of the text. Then our
method computes similarities between the text and the selected clusters using
common features. Following that it assigns the text to an appropriate cluster
(new or existing) using the dynamically computed similarity thresholds based
on statistical measure. After that it builds a clustering model using the
cluster assignment of the text to reflect the addition of this text to a new
or an existing cluster; and uses the current clustering model to cluster the
subsequent text. The details are shown in Algorithm 1 and are described next.

Algorithm 1 Proposed Text Stream Clustering

Input: Texts: $t_{1}...t_{\infty}$, $DI$: Delete-interval
Output: Cluster assignments: $z_{t_{1}...t_{\infty}}$

1: for $t_{i}$ in $t_{1}...t_{\infty}$ do

2: Extract features ($f$) from $t_{i}$

3: Select $L$ clusters that share common features with $t_{i}$ (described in
Section 4.1)

4: Compute similarities ($s_{l}$) between $t_{i}$ and the $L$ selected
clusters

5: Compute the maximum ($\max_{l}$), mean ($\mu_{l}$) and standard deviation
($\sigma_{l}$) of the $s_{l}$ similarities

6: if $\max_{l}>\mu_{l}+\sigma_{l}$ then

7: $j=\,$ cluster index for $\max_{l}$

8: Assign $t_{i}$ to $j^{th}$ cluster

9: else

10: Assign $t_{i}$ to a new cluster

11: end if

12: Build clustering model (described in Section 4.4)

13: if $i\mod DI=0$ then

14: Delete outdated clusters (described in Section 4.5)

15: end if

16: end for

###  4.1. Selecting Clusters for the Text

For each short text, we select a specific set of clusters that share common
features with the text. For each feature in a text, we obtain the cluster
$ids$ from the corresponding feature vector $F$. We combine the cluster $ids$
obtained using the features in a text. These are the selected clusters that
share common features with the text.

###  4.2. Computing Similarities between the Text and Selected Clusters

FastStream computes similarities between the text and the selected clusters
based on the common features as shown in Equation 1.

(1) $similarity(t,z)=\frac{2\times\sum_{f\in
t}\min(n_{z}^{f},N_{t}^{f})}{N_{t}+n_{z}}$

To compute similarity between a text $t$ and a cluster $z$, we sum the
occurrences of the common features between $t$ and $z$ which is then
normalized by the summation of the total number of features in text $t$ and
cluster $z$ denoted by $N_{t}$ and $n_{z}$ respectively. Here $n_{z}^{f}$,
$N_{t}^{f}$ refer to the features ($f$) along with frequencies in cluster $z$
and text $t$ respectively.

###  4.3. Assigning Text to Cluster

To assign a text to a cluster, we compute the maximum ($max$), mean ($\mu$)
and standard deviation ($\sigma$) of the similarities between the text $t$ and
the selected clusters. We assign the text to the cluster with the maximum
similarity if the maximum similarity is greater than the $\mu+\sigma$ of the
similarities. Otherwise, we create a new cluster containing this text. Thus
the texts are assigned to clusters based on the dynamically computed
similarity thresholds. The intuition behind using maximum similarity greater
than $\mu+\sigma$ is that, this maximum similarity is above the average
similarities reflecting that both the text and the target cluster share highly
similar content.

###  4.4. Building Clustering Model

We build clustering model using the cluster assignment of the text to reflect
the addition of this text to an existing or a new cluster. When a text $t$ is
added to a cluster $z$, we update the corresponding $CF$ vector by updating
its features with frequencies ($n_{z}^{f}$), number of features ($n_{z}$), and
number of texts ($m_{z}$). The addible property of the $CF$ vector is
described in the following.

##### Addible Property of Text to Cluster:

$\\\ n_{z}^{f}=n_{z}^{f}+N_{t}^{f}\;\;\;\forall f\in\mathbb{\,}t\\\
n_{z}=n_{z}+N_{t}\\\ m_{z}=m_{z}+1\\\ id_{z}=\begin{cases}id_{z},\;\text{if
successive texts are in same cluster}\\\ max(id_{z})+1\;\;\;\forall
z\in\mathbb{\,}Z,\;\text{otherwise}\end{cases}\\\ \\\ $

Here, $N_{t}^{f}$ and $N_{t}$ refer to the features with frequencies in text
$t$ and the total number of features in text $t$ respectively. $Z$ is the set
of all $CF$ vectors (i.e., $Z=\\{CF\\}$).

A new $id$ is assigned to the cluster $z$ (new or existing) if the cluster
assignment of the current text is different from that of the previous text,
otherwise the cluster $id$ of the current text remains same as that of the
previous text. This implies that the recently created or updated cluster will
have the highest cluster $id$. For each feature in the text, we append the
cluster $id$ to the corresponding feature vector $F$.

###  4.5. Deleting Outdated Clusters

We remove the outdated clusters based on their update-timestamps (represented
by cluster $id$) and cluster-sizes (i.e., number of texts in clusters). The
recently created or updated clusters will have higher cluster $ids$.

We remove outdated clusters in every Delete-interval. Delete-interval is the
interval when we remove outdated clusters after clustering a certain number of
texts. For example, the Delete-interval equal to 500 implies that we delete
outdated clusters after we cluster every 500 texts. To obtain outdated
clusters, we calculate the $\mu$ and $\sigma$ of the cluster $ids$ and
cluster-sizes in every Delete-interval. If the _cluster id_ is less than the
$\mu-\sigma$ of cluster $ids$ and the cluster-size is less than the
$\mu-\sigma$ of cluster-sizes, then we delete the cluster by deleting the
corresponding $CF$ vector and remove the corresponding cluster $id$ from the
feature vectors ($F$) that contain that particular cluster $id$.

##  5\. Experimental Study

###  5.1. Datasets

We used four different datasets of short texts in our experiments. The basic
properties of these datasets are shown in Table 1.

Table 1. Summary of the short text datasets Dataset | #Clusters | #Texts | Avg. #words/text
---|---|---|---
Ns-T | 152 | 11,109 | 6.23
Ts-T | 269 | 30,322 | 7.97
SO-T | 10,573 | 1,23,342 | 5.57
NTSO-T | 10,994 | 1,64,773 | 6.02

The datasets Ns-T (Yin et al., 2018) and Ts-T (Yin et al., 2018) consist of
11,109 news titles and 30,322 tweets and are distributed into 152 and 269
groups respectively. We create a new dataset called SO-T using the titles of
the StackoverFlow questions consisting of 1,23,342 question titles distributed
into 10,573 clusters. We combined the texts of the datasets Ns-T, Ts-T, and
SO-T and created a combined dataset NTSO-T consisting of 1,64,773 texts
distributed into 10,994 clusters. The average number of words per text in the
datasets Ns-T, Ts-T, SO-T, and NTSO-T are 6.23, 7.97, 5.57, and 6.02
respectively. The texts in these four datasets were randomly shuffled to
examine how FastStream and the state-of-the-art methods (MStream(F) (Yin et
al., 2018), DP-BMM(-FP) (Chen et al., 2020), and OSDM (Kumar et al., 2020))
perform when dealing with the texts from different domains arriving in random
order.

####  5.1.1. Construction of the Dataset SO-T

We create a dataset SO-T using the titles of the duplicate questions posted in
StackOverflow website333https://stackoverflow.com/ on various topics such as
_Java_ , _Python_ , _JQuery_ , _R_ , and so on. We consider that duplicate
questions are similar to each other and a group of similar questions can form
a cluster.

We obtain the question titles from the file
Posts.xml444https://meta.stackexchange.com/questions/2677/database-schema-
documentation-for-the-public-data-dump-and-sede and obtain the information
about the duplicate questions from PostLinks.xml. Each item in Posts.xml
represents a single post which can be of different types (e.g., question,
answer, and so on). Each item of the PostLinks.xml contains the information
about a pair of duplicate questions. For instance, the PostLinks.xml contains
the questions A and B if they are duplicate. There are 20,094,655 questions in
Posts.xml and 1,009,249 pair of duplicate questions in PostLinks.xml. Among
the 1,009,249 pair of duplicate questions, we randomly select 400,000
pairs555We select this specific number of pairs (400,000) because of the
maximum capacity of the computer (Core i5-4200U and 8GB memory) where the
experiments were carried out..

Using the duplicate information in PostLinks.xml, we create a list of directed
edges (e.g., $A\rightarrow B$, $B\rightarrow C$) which are then used to create
a graph. To obtain the clusters of duplicate questions, we compute connected
components666We use the library in https://networkx.github.io/ to compute
connected components. using the representation of the graph. In particular, If
A and B are duplicate, and B and C are duplicate, then we obtain the connected
component $A\rightarrow B\rightarrow C$ which is considered as a cluster of
the duplicate questions of A, B, and C.

We compute the length of the connected components defined as the number of
questions in the component. After that, we compute the mean ($\mu$) and
standard deviation ($\sigma$) of the lengths of the connected components and
select the components whose lengths are between the $\mu\pm\sigma$ which in
turn produces 10,573 connected components (i.e., clusters) consisting of
1,23,342 question titles. Sample StackOverflow question titles (with _PostId_)
of a cluster in the dataset SO-T are shown in Figure 1.

– Python Video Framework (1003376)
– Best video manipulation library for Python? (220866)
– Trim (remove frames from) a video using Python (7291653)

Figure 1. Sample StackOverflow question titles (with _PostId_) of a cluster in
the dataset SO-T.

###  5.2. Proposed Clustering Method with Different Types of Features

We represent each text using three different types of features which are
unigram, bigram and biterm and use each representation separately to cluster
the streams of texts. When we use unigram, bigram or biterm feature, our
method is called as FastStream-unigram, FastStream-bigram, and FastStream-
biterm respectively.

###  5.3. Optimal Delete-interval for Proposed Method

Our proposed method (FastStream) requires only one parameter called Delete-
interval ($DI$) to delete the outdated clusters. We set Delete-interval to 500
for all the datasets used in our experiments. How we choose this value is
discussed in the following.

Delete-interval is the interval when we remove outdated clusters after
clustering a certain number of texts. To determine the optimal value of
Delete-interval, we choose the dataset Ns-T and represent the text using
biterm features. We determine the value of Delete-interval based on the
optimal clustering performance of our method for the dataset Ns-T; and use
this value to delete outdated clusters for other datasets. The clustering
performance (in terms of NMI) of our method for different Delete-intervals on
the dataset Ns-T is shown in Figure 2.

$100$$300$$500$$700$$900$$70$$75$$80$$85$$90$Delete-intervals ($DI$)NMI (%)NMI
for Ns-T Figure 2. NMI results of our short text stream clustering method for
different Delete-intervals ($DI$) on dataset Ns-T.

Based on the different values of $DI$, we observe that we achieve the highest
NMI for Ns-T when $DI$=500. Therefore, we choose $DI$=500 for all datasets
used in our experiments to delete outdated clusters.

###  5.4. Baseline Clustering Methods

We compare the performance of FastStream with recent state-of-the-art short
text stream clustering methods as described in the following.

  * •

MStream (Yin et al., 2018) algorithm clusters each batch of short texts at a
time. It stores all the clusters produced over the course of time. MStream has
one pass clustering process and update clustering process of each batch. In
update clustering process, it applies gibbs sampling to the same batch of
texts multiple times to improve the initial clustering result obtained in one
pass clustering process.

  * •

MStreamF (Yin et al., 2018) is a variant of MStream algorithm that deletes the
outdated clusters of previous batches and only stores the clusters of the
current batch.

  * •

DP-BMM (Chen et al., 2020) is a short text stream clustering algorithm that
adopts the similar approach as MStream and clusters each batch of short texts
at a time as it arrives. The principal difference between DP-BMM and MStream
is that DP-BMM represents the texts using biterm features instead of unigrams
(i.e., words). Therefore, DP-BMM stores the clusters of texts using biterm
features.

  * •

DP-BMM-FP (Chen et al., 2020) is a variant of DP-BMM algorithm that deletes
the outdated clusters of previous batches and only stores the clusters of the
current batch similar to MStreamF.

  * •

Rakib et al. (2020) proposed a short text stream clustering method by adopting
the clustering model of MStream algorithm (Yin et al., 2018). The main
difference between this method and MStream algorithm is that this method
removes outliers from the clusters obtained by MStream algorithm and reassigns
the outliers to the clusters using dynamically computed similarity thresholds.

  * •

OSDM (Kumar et al., 2020) is a short text stream clustering algorithm that
clusters each short text one by one as it arrives. OSDM deletes a cluster if a
cluster becomes outdated, that is, the cluster is not being updated for a
while over time.

For MStream(F), we set the parameters $\alpha=0.03$ and $\beta=0.03$ for all
the datasets as defined in (Yin et al., 2018). Likewise, for DP-BMM(-FP), we
set $\alpha=0.6$ and $\beta=0.02$ as mentioned in (Chen et al., 2020).
MStream(F) and DP-BMM(-FP) cluster each batch of texts at a time. We set the
batch size777The batch size equal to 2000 was chosen for MStream(F) and DP-
BMM(-FP) based on their optimal performance on the datasets used in this
paper. to 2000 for MStream(F) and DP-BMM(-FP) for all datasets. We set the
number of iterations to 10 for MStream(F) and DP-BMM(-FP), and number of saved
batches to one for MStreamF and DP-BMM-FP as mentioned in (Chen et al., 2020).

For OSDM, we set $\alpha=2e^{-3}$, $\beta=4e^{-5}$, and an additional
parameter $\lambda=6e^{-6}$ for all the datasets as defined in (Kumar et al.,
2020). OSDM used $\lambda$ as the decay rate which is used to set lower
weights to the clusters which are not being updated over time and need to be
deleted in the future.

###  5.5. Faster Version of Baseline Clustering Methods

  * •

Fast-MStream888https://github.com/rashadulrakib/short-text-stream-
clustering/tree/master/FastBatchClustering and Fast-MStreamF are the faster
versions of MStream and MStreamF algorithms that we developed respectively. We
apply the inverted index (Ilic et al., 2014) based searching technique using
the words of the clusters produced by the MStream and MStreamF algorithms.

  * •

Fast-Rakib et al. (2020) is the faster version of Rakib et al. (2020). We
index the clusters produced by Rakib et al. (2020) using unigrams following
the notion of Fast-MStream.

###  5.6. Comparison with State-of-the-art Methods

####  5.6.1. Comparison of Clustering Results

We compare the performance of our proposed method (FastStream) and the
proposed faster versions of the state-of-the-art short text stream clustering
methods with the state-of-the-art methods. We apply different types of text
representations (unigram, bigram, and biterm) to FastStream denoted as
FastStream-unigram, FastStream-bigram, and FastStream-biterm. We use
normalized mutual information (NMI) as the evaluation measures for evaluating
the performance of different clustering methods. We randomly shuffle each
dataset 20 times. Then we perform 20 independent trials for each of the
methods on each dataset. The average NMI results999We could not run DP-
BMM(-FP) on the datasets SO-T and NTSO-T because of their longer running time
that exceeds the capacity of the experimental computer. of these runs are
shown in Table 2.

Table 2. Normalized Mutual Information (NMI) score of different clustering methods. The highest result for a particular dataset is denoted bold. Clustering | Eva. | Data Sets
---|---|---
Methods |  | Ns-T | Ts-T | SO-T | NTSO-T
FastStream-unigram |  | 0.829 | 0.831 | 0.732 | 0.725
FastStream-bigram |  | 0.837 | 0.826 | 0.754 | 0.729
FastStream-biterm |  | 0.844 | 0.848 | 0.777 | 0.774
Fast-MStream |  | 0.857 | 0.868 | 0.615 | 0.611
Fast-MStreamF |  | 0.877 | 0.878 | 0.651 | 0.613
Fast-Rakib et al. (2020) | NMI | 0.864 | 0.901 | 0.654 | 0.639
MStream |  | 0.859 | 0.867 | 0.618 | 0.608
MStreamF |  | 0.879 | 0.877 | 0.652 | 0.619
DP-BMM |  | 0.883 | 0.862 | – | –
DP-BMM-FP |  | 0.838 | 0.875 | – | –
OSDM |  | 0.858 | 0.842 | 0.463 | 0.441
Rakib et al. (2020) |  | 0.862 | 0.892 | 0.658 | 0.641

Our experimental results show that FastStream (-unigram,-bigram, and -biterm)
perform significantly better than the state-of-the methods in terms of NMI on
the dataset SO-T and the combined dataset NTSO-T. Among the three variants of
FastStream, FastStream-biterm performs better than FastStream-unigram and
FastStream-bigram on all the datasets. The reason is that, by using biterm
features, we can extract sufficient contexts for short texts and the biterm
features are more distinctive which in turn more likely to cluster the texts
into their proper clusters (Chen et al., 2020).

The state-of-the-art methods MStream(F), DP-BMM(-FP), and OSDM perform better
than FastStream on the datasets Ns-T and Ts-T since these methods tune the
hyper parameters (e.g., $\alpha$, $\beta$) on these datasets to obtain optimal
clustering performance. On the contrary, our method uses dynamic similarity
thresholds to assign texts to the clusters (new or existing) and does not
require this kind of hyper parameter tuning on a particular dataset.

The overall performance of FastStream is comparable to the performance of the
state-of-the-art methods on the datasets Ns-T and Ts-T. In addition, our
method significantly outperforms the state-of-the-art methods on the datasets
SO-T and NTSO-T as the hyper parameters of the state-of-the-art methods were
not tuned on these two datasets.

####  5.6.2. Comparison of Running Time

The average running times (in seconds) of FastStream and the state-of-the-art
methods are shown in Table 3.

Table 3. Average running times (in seconds) of different methods. Clustering | Data Sets
---|---
Methods | Ns-T | Ts-T | SO-T | NTSO-T
FastStream-unigram | 3 | 20 | 85 | 168
FastStream-bigram | 3 | 18 | 86 | 167
FastStream-biterm | 4 | 24 | 101 | 183
Fast-MStream | 115 | 219 | 1318 | 1790
Fast-MStreamF | 83 | 193 | 811 | 1198
Fast-Rakib et al. (2020) | 32 | 79 | 479 | 613
MStream | 227 | 541 | 3319 | 4289
MStreamF | 173 | 295 | 1481 | 2137
DP-BMM | 5016 | 12307 | – | –
DP-BMM-FP | 880 | 2854 | – | –
OSDM | 31 | 196 | 1103 | 1882
Rakib et al. (2020) | 80 | 165 | 889 | 1389

The running time of FastStream is several orders of magnitude faster than that
of MStream(F), DP-BMM(-FP), and OSDM on all datasets. In addition, we improve
the running time of the existing methods (e.g., Fast-MStream, Fast-MStreamF,
and Fast-Rakib et al. (2020)) by using inverted index (Ilic et al., 2014). The
reason is that we do not compute similarity between a text and all clusters
while assigning a text to a cluster. Instead we select a specific set of
clusters using the biterms of the text based on inverted index and compute
similarities between the text and the selected clusters. We store almost twice
the number of biterms than other methods as we use inverted index to select a
specific set of clusters for a particular text. We consider this as a small
price to pay for the significant improvement in running time of our proposed
method (FastStream) and the existing state-of-the-art methods.

##  6\. Conclusion and Future Work

We have demonstrated that building an efficient clustering model based on
inverted index and by assigning texts to the clusters using dynamic similarity
thresholds improves the clustering quality of our proposed method and
outperforms the state-of-the-art short text stream clustering methods on
larger datasets.

We also demonstrated that by computing similarity between a text and a
specific set of clusters instead of all clusters, we significantly reduce the
running time of our method (FastStream) and the faster versions of the state-
of-the-art methods than that of the existing state-of-the-art methods. We
contribute two new datasets SO-T and NTSO-T comprising of texts from various
domains to examine how the FastStream and other state-of-the-art methods
perform when texts from different domains arrive in random order.

In the future, we plan to cluster significantly larger short text streams
(composed of a few million texts). We also plan to apply our improved text
stream clustering algorithm to better identify groups of users of social media
platforms interested in similar topics.

## References

  * (1)
  * Aggarwal et al. (2003) Charu C. Aggarwal, Jiawei Han, Jianyong Wang, and Philip S. Yu. 2003\.  A Framework for Clustering Evolving Data Streams. In _Proceedings of the 29th International Conference on Very Large Data Bases_ (Berlin, Germany). 81–92.
  * Blei and Lafferty (2006) David M. Blei and John D. Lafferty. 2006.  Dynamic Topic Models. In _Proceedings of the 23rd International Conference on Machine Learning_ (Pittsburgh, Pennsylvania, USA). ACM, New York, NY, USA, 113–120.
  * Carnein and Trautmann (2019) Matthias Carnein and Heike Trautmann. 2019.  Optimizing Data Stream Representation: An Extensive Survey on Stream Clustering Algorithms.  _Business & Information Systems Engineering_ 61, 3 (01 Jun 2019), 277–297.
  * Chen et al. (2020) Junyang Chen, Zhiguo Gong, and Weiwen Liu. 2020.  A Dirichlet process biterm-based mixture model for short text stream clustering.  _Applied Intelligence_ 50, 5 (2020), 1609–1619.
  * Erk (2012) Katrin Erk. 2012\.  Vector Space Models of Word Meaning and Phrase Meaning: A Survey.  _Language and Linguistics Compass_ 6 (2012), 635–653.
  * Ilic et al. (2014) M. Ilic, P. Spalevic, and M. Veinovic. 2014.  Inverted index search in data mining. In _2014 22nd Telecommunications Forum Telfor (TELFOR)_. 943–946.
  * Ishwaran and James (2001) Hemant Ishwaran and Lancelot F. James. 2001.  Gibbs Sampling Methods for Stick-Breaking Priors.  _J. Amer. Statist. Assoc._ 96, 453 (1 3 2001), 161–173.
  * Kalogeratos et al. (2016) Argyris Kalogeratos, Panagiotis Zagorisios, and Aristidis Likas. 2016.  Improving Text Stream Clustering Using Term Burstiness and Co-Burstiness. In _Proceedings of the 9th Hellenic Conference on Artificial Intelligence_ (Thessaloniki, Greece) _(SETN ’16)_. Association for Computing Machinery, New York, NY, USA, Article 16, 9 pages.
  * Kumar et al. (2020) Jay Kumar, Junming Shao, Salah Uddin, and Wazir Ali. 2020\.  An Online Semantic-enhanced Dirichlet Model for Short Text Stream Clustering. In _Proc. of the 58th Annual Meeting of the Association for Computational Linguistics_. Online, 766–776.
  * Liang et al. (2016) Shangsong Liang, Emine Yilmaz, and Evangelos Kanoulas. 2016\.  Dynamic Clustering of Streaming Short Documents. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (San Francisco, California, USA). 995–1004.
  * Rakib et al. (2020) Md Rashadul Hasan Rakib, Norbert Zeh, and Evangelos Milios. 2020\.  Short Text Stream Clustering via Frequent Word Pairs and Reassignment of Outliers to Clusters _(DocEng ’20)_. Association for Computing Machinery, New York, NY, USA, Article 13, 4 pages.
  * Shou et al. (2013) Lidan Shou, Zhenhua Wang, Ke Chen, and Gang Chen. 2013\.  Sumblr: Continuous Summarization of Evolving Tweet Streams. In _Proceedings of the 36th International ACM SIGIR Conference on Information Retrieval_ (Dublin, Ireland). 533–542.
  * Yin et al. (2018) Jianhua Yin, Daren Chao, Zhongkun Liu, Wei Zhang, Xiaohui Yu, and Jianyong Wang. 2018\.  Model-based Clustering of Short Text Streams. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (London, United Kingdom). 2634–2642.
  * Zhao et al. (2016) Yukun Zhao, Shangsong Liang, Zhaochun Ren, Jun Ma, Emine Yilmaz, and Maarten de Rijke. 2016\.  Explainable User Clustering in Short Text Streams. In _Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Pisa, Italy). ACM, 155–164.