# TransPOS: Transformers for Consolidating Different POS Tagset Datasets

Alex Li1, Ilyas Bankole-Hameed1, Ranadeep Singh1, Gabriel Shen Han Ng1, Akshat
Gupta2
1Carnegie Mellon University, 2JPMorgan AI Research, New York, USA
{alexli2, ibankole, ranadees<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

In hope of expanding training data, researchers often want to merge two or
more datasets that are created using different labeling schemes. This paper
considers two datasets that label part-of-speech (POS) tags under different
tagging schemes and leverage the supervised labels of one dataset to help
generate labels for the other dataset. This paper further discusses the
theoretical difficulties of this approach and proposes a novel supervised
architecture employing Transformers to tackle the problem of consolidating two
completely disjoint datasets. The results diverge from initial expectations
and discourage exploration into the use of disjoint labels to consolidate
datasets with different labels.

TransPOS: Transformers for Consolidating Different POS Tagset Datasets

Alex Li1, Ilyas Bankole-Hameed1, Ranadeep Singh1, Gabriel Shen Han Ng1, Akshat
Gupta2 1Carnegie Mellon University, 2JPMorgan AI Research, New York, USA
{alexli2, ibankole, ranadees<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

There has been an explosion in the availability and variety of labeled
datasets in almost every domain. Unfortunately, Artificial Intelligence (AI)
practitioners and researchers often find themselves unable to make use of
labeled datasets for tasks related but not identical to their tasks. This is
primarily due to different labeling schemes where a trivial mapping to merge
the datasets into one larger dataset does not exist. In this paper, we explore
the possibility of consolidating datasets that were curated for the same task
with different labeling schemes. To make this easy to apply to any pair of
datasets, we consider a very interesting scenario in which we attempt to make
a model that can understand both datasets without ever actually seeing any
examples that have labels from both of them.

There are several domains and application areas to which our technique can be
applied to, and frankly might be the only option. For example: When creating a
data set to detect people, objects, and vehicles in an urban environment, we
may want to supplement our existing data set with the popular Cityscapes
dataset (Cordts et al., 2016), but struggle to directly apply those labels to
the merged dataset due to a few minor differences in the label scheme, such as
a smaller or larger label set. There could also be some information partially
correlated with the existing dataset’s labels; perhaps in our dataset we have
to distinguish between standing and sitting people. Cityscapes does not
distinguish between these, so is it possible to use the label information
(about where humans are) to get high-quality segmentation under our desired
labeling scheme?

The focus of this paper comes from part-of-speech (POS) tagging. Although some
tags are common to all datasets, different datasets may have different
conventions for how to deal with more uncommon parts of speech, like modal
verbs, particles, or even when to treat something as a noun. These problems
are exacerbated in informal contexts. We provide a novel design for a
supervised model that can translate labels from one dataset into another
labeled dataset without requiring any shared examples. After analyzing
results, we reconsider the situations under which it is possible to squeeze
out extra performance from these labels, and show that it is unlikely for any
kind of architecture to use label information to perform better than an
equivalent model that does not, unless the architecture has access to shared
examples or metadata about the meaning of the labels.

###  1.1 Related Work

The problem of dissimilar POS tagsets has historically been approached in two
significant ways:

  1. 1.

Supervised Learning: (Shen, 2007) proposed a supervised POS tagger with 97.3%
accuracy for the English language;

  2. 2.

Create Dictionary Mapping: (Petrov et al., 2011) proposed a Universal POS
tagset to map 25 different treebank tagsets to 12 universal POS tags.

There has also been a significant amount of progress in creating POS tags for
languages other than English leveraging both supervised and unsupervised
methods (Das and Petrov, 2011).

Our work can be seen as a type of Multitask Learning (Caruana, 1998) as we are
learning from two related datasets that have been labeled independently and
differently. A common technique is to create a model for each task (Collobert
and Weston, 2008), and enforce weight sharing between their lower layers to
allow shared low-level domain knowledge. A key distinction between our
methodology and Multitask Learning is that our test time goal also makes use
of labels from the other task. We use the actual predictions of the model
rather than the more common idea of using the predicted logits or encoded
representation from a previous layer.

This problem can also be considered as a type of Domain Adaptation Technique.
However, many domain adaptation algorithms ((Daumé, 2009)) assume some shared
examples between the source and target domains, so we cannot apply it in our
case. Those algorithms that do not make this assumption have never to our
knowlege tried to use the labels that are in the target distribution but are
not the source distribution labels.

##  2 Setting

Let $\Sigma$ be the set of unicode characters. In our setting, we have two
datasets that map from the space of sentences of unicode characters
$X=\bigcup\Sigma^{n}$ to part-of-speech tags. However, the two datasets use
different labeling schemes: the first may use the standard Universal POS
tagset $Y$ while the second uses a proprietary POS tagset $Z$. Each sentence
in the first dataset has a label for each word in
$\bigcup_{n=0}^{\infty}Y=\mathbf{Y}$, while each sentence in the second has a
label in $\bigcup_{n=0}^{\infty}Z=\mathbf{Z}.$

Then we can name the two datasets as
$\mathcal{D}_{Y}=\\{(x_{y}^{(i)},y^{(i)})\in(X,\mathbf{Y})\\}_{i}$ and
$\mathcal{D}_{Z}=\\{(x_{z}^{(i)},z^{(i)})\in(X,\mathbf{Z})\\}_{i}$.
Presumably, $\mathbf{Y}$ and $\mathbf{Z}$ are very highly correlated, since
they are both POS tags for a sentence, just defined with slightly different
rules. We would like to expand the dataset $\mathcal{D}_{Y}$ to include the
sentences and labels of $\mathcal{D}_{Z}$, but unfortunately the labels are
incompatible. However, we expect that we can still get useful information from
the labels $\mathbf{Z}$. Therefore, our goal is to build a predictor function
$f_{Y}:(X,\mathbf{Z})\to\mathbf{Y}$ that combines both the text and the
information of the annotated $\mathcal{D}_{Z}$ to predict what the translated
label would be in the tagset $\mathbf{Y}$. Similarly, we consider the
construction of $f_{Z}:(X,\mathbf{Y})\to\mathbf{Z}$.

Here are the two obvious baselines that could be used to construct $f_{Y}$:

  1. 1.

Direct Map We could use domain knowledge to directly design a mapping from
each label $Z\to Y$. If $|Z|<|Y|$, $Y$ is not a deterministic predictor of $Z$
and this introduces noise into the system. If $|Z|>|Y|$, converting to $Y$
will result in a loss of information.

  2. 2.

Supervised Model We could train a model on $\mathcal{D}_{Y}$ to build a
function $X\to\mathbf{Y}$.

Note that while the second baseline is trained with data, the first baseline
is completely based on human understanding of the relationship between labels.
Thus, while we can naturally train a model to match the performance of the
Supervised Model, it is much less obvious how we can train a model to gain the
performance advantage given by the Direct Map method.

Now we consider the design of our model intended to use information about both
X and $\mathbf{Z}$ to perform better than either approach.

##  3 Model

In our method, we will transform our input $X$ into an embedding space $E$
using a transformer’s encoder $Enc:X\to E$ and two GRU decoder functions, one
for each type of label $\mathbf{Y}$ and $\mathbf{Z}$.
$D_{Y}:(E,\mathbf{Z})\to\mathbf{Y}$ and $D_{Z}:(E,\mathbf{Y})\to\mathbf{Z}$.
Then to infer a label Z for a given training sample $(x_{y},y)$, we can
compute $D_{Z}(Enc(x_{y}),y).$ See Figure 1 for a visualization.

$x_{y}$$y$$x_{z}$$z$$e_{y}=Enc(x_{y})$$e_{z}=Enc(x_{z})$$\tilde{z}=D_{Z}(e_{y},y)$$\tilde{y}=D_{Y}(e_{z},z)$

Figure 1: When evaluating the model, we use $y$ and $z$ as inputs!

$x_{y}$$P(z|x_{y})=S_{Z}(e_{y})$$e_{y}=Enc(x_{y})$$s_{y}=S_{Y}(e_{y})$$\hat{z}\sim
P(z|x_{y})$$\tilde{y}=D_{Y}(e_{y},\hat{z})$$Loss=CE(y,\tilde{y})+CE(y,s_{y})$

Figure 2: To train the model with the $Y$ dataset, we simulate having Z labels
by sampling from the logits of a normal supervised model $S_{Z}$. At the same
time, we train a model $S_{Y}$ for the other dataset. Heavy dropout is applied
at the location of the blue arrow.

However, the setup used for validation will not work for training the model.
Ideally, we would like to make a loss function that penalizes the predicted
value of $z$ from being far from the true $z$ corresponding to $x_{y}$, but we
do not have any access to the true z! We only have pairs $(x,y)$ and $(x,z)$,
not $(x,y,z)$. One way to fix this is to first predict $\hat{z}$ using $(x,y)$
and then use that as a surrogate for true $z$. To do this, we can create a
supervised model $S_{Z}$ that takes in the encoded variables $Enc(x_{y})$ and
outputs a prediction for the label $z$, which we can then input into the
decoder $D_{Z}.$ However, the careful reader will notice a flaw in this
strategy: While $D_{Z}$ takes the input as one-hot labels at inference time,
it takes input as logits at training time. To reconcile this difference, we
treat the softmax of the predicted logits as a probability distribution, from
which we sample our true predicted label $\hat{z}$.

The entire training process with the inputs $\mathcal{D}_{Y}$ is shown in
Figure 2, and the model for $\mathcal{D}_{Z}$ is made the same way, but with
the y / z inputs flipped. In our implementation, each mini-batch contains some
examples from both datasets. To reduce the complexity of the model, we use the
same base encoder model weights for $S_{Y},S_{Z},D_{Y},$ and $D_{Z}$, though
in principal they could be different or only partially shared.

The inquisitive reader may wonder why we use $S_{Z}$ to predict labels $z$
instead of reusing labels $D_{Z}$ along with ground truth labels $y$. In this
case, we will have given the label that we want to predict as an input to the
model, and so the model can simply learn to predict the input! For example,
suppose that in our model, rather than sampling $\hat{z}\sim S_{Z}(e_{y})$, we
reused the decoder weights to get $\hat{z}:=D_{Z}(e_{y},y)$, then predicted
$\tilde{y}=D_{Y}(e_{y},\hat{z})$ (and similarly on the other side). Then, the
model can simply learn to ignore the first argument of $D_{Y}$ and $D_{Z}$ and
instead learn that $D_{Y}(-,z)$ and $D_{Z}(-,y)$ are inverses to each other.
In this setup, it will perfectly predict all the training data, but it will be
completely useless in practice. We actually tried this setup and found that
the model would actually achieve performance competitive with the supervised
model for a few epochs (perhaps due to regularization like dropout), but after
training long enough, it learns the cheat and arrives at 0 training error and
very high validation error. Now, during training, $y$ and $z$ are completely
derived from $x$. So, in principle, $D_{Y}$ and $D_{Z}$ may learn to ignore
noisy outputs $y$ and $z$ and make predictions based solely on $x$. To prevent
this, we enforce a very heavy dropout of $0.85$ on the first term before
passing it as input.

The model and training code can be found in out Github
repository111https://github.com/Alex7Li/TransPOS in the footnote.

##  4 Datasets

Text | Ark Label
---|---
New | Adjective
FC | Proper noun
Menu | Proper noun
Utility | Proper noun
2.0 | numeral
#apple | Proper noun
http://t.co/VftFt2c | URL or email address
Text | Tweebank Label
@USER2082 | A
good | ADJ
night | NOUN
I | PRON
Love | VERB
You | PRON
:) | SYM
http://t.co/VftFt2c | U
Table 1: Example tweets from Ark and Tweebank

In this project, we consider two datasets:

  1. 1.

ARK-Twitter Kevin Gimpel (2011), which contains 34k tokens from tweets sampled
primarily on Oct 27, 2010.

  2. 2.

Tweebank dataset Yijia Liu et al. (2018) which maintains 840 tweets from
Tweebank v1, 2500 examples from twitter stream from February 2016 to July
2016.

The Tweebank dataset used UD annotation conventions, while the ARK data set
used the Stanford POS Tagger trained in WSJ.

However; these two datasets have a data contamination problem: there are 210
identical shared tweets. In our case, however, this served as the perfect
validation set for our model.

Looking at the distribution of the labels in this validation set (Figure 3),
we see that the ambiguity between the meanings of the labels will limit the
performance of a direct mapping.

##  5 Experiments

Figure 3: Distribution of validation labels.
GPT-2 | Tweebank Acc | Ark Acc
---|---|---
supervised model |  89.53% | 89.92%
our model |  89.53% |  90.17%
supervisor only | 86.96% | 88.29%
no label input | 88.09% | 88.59%
Bertweet-large | Tweebank Acc | Ark Acc
supervised model |  94.31% | 95.02%
our model | 94.26% | 94.97%
supervisor only |  94.31% |  95.09%
no label input | 94.22% | 94.97%
direct map | 88.31% | 89.97%
Table 2: Accuracy (Acc) with Bertweet-large model baseline

The first baseline was created by making a ‘direct map’ between the labels. We
looked at the validation set and chose the map that gave the highest possible
score.

The second ’supervised model’ baseline was a normal transformer model; we
train on the train split of one dataset and evaluate on the validation split
of that same dataset.

Then, for ‘our model’, we trained with the described architecture, using the
transformers BERTweet (Nguyen (2020)), and GPT-2 (Radford et al. (2019)) a
GPT-2 model and a Bertweet model with the described architecture with dropout
$.85$. After the training was complete, we evaluated the accuracy using the
method described above to compute our model accuracy.

To see if our model was really learning from the y labels, we used of the
supervised model heads $S_{Y}\circ Enc$ and $S_{Z}\circ Enc$ to get the
‘supervisor only’ accuracy. This architecture is exactly the same as the
baseline, but differences arise in the accuracy because the training process
is not the same (in particular, there is very high $x$ dropout).

Finally, we considered the accuracy of the full pipeline when there with ‘no
label input’: instead of providing the $z$ labels for the first dataset while
predicting the $y$ labels of the second, we just took the $z$ labels that the
model predicted and sampled from that distribution as we do at training time.

##  6 Results

Figure 4: GPT-2 Validation accuracy

We trained the model with both the GPT-2 as our encoder and the Bertweet model
as our encoder. All models were trained for 25 epochs, and we report the
accuracy at the final epoch in Table 2.

In both cases, our accuracy does not exceed the baseline. Although the
Bertweet model appears to gain nothing from the $z$ labels, the GPT-2 model
appears to be using the $z$ labels to effectively improve performance as
indicated in Figure 4. Since the score of the model improves when we give it
the z labels, we can say that it is actually learning to use the joint
probability distribution of the $y$ and $z$ labels.

Our approach can only be useful when the correlation between the labels of
both datasets provides information that the correlation between the encoder
output and the true labels does not. One possible explanation for our
inability to beat the baseline model is because the label information was not
sufficient or because the baseline model was already too strong. However,
using the weaker GPT-2 model as a baseline did not show any improvement.

Although there are a multitude of different ideas for model designs that use
the $y$ labels, it is important to first try and understand why this model
struggled in this regime. From our results, it seems it will be difficult to
design an architecture that can effectively learn from the label information
of another dataset without using any shared examples.

To emphasize that this problem will be hard for any architecture, let us
consider a toy example of this problem where we no longer have any $X$ data
and are just given a set of $\mathbf{Y}$ POS tokens and $\mathbf{Z}$ POS
tokens. In this case, the the $y$ and $z$ labels are still very correlated,
but since it is impossible for a model to predict the price from an integer
id, our model will not be able to learn about and make use of the high
covariance between labels. As we have given the model two unrelated sets of
labels, no matter what model you use, it will be impossible to relate them
with anything other than the statistical properties of the $\mathbf{Y}$ and
$\mathbf{Z}$ distributions. This does not seem too informative in general,
since it will be difficult to find the correct relationship between two sets
with no shared examples, though the fact that POS is a multi-label prediction
problem means that you might be able to get a bit out of it. Still, even
trying to make the label distributions similar is not easy as the labels are
not in the same space.

In our model, we consider pairs of predicted $y$ and true $z$ data, which
ultimately cannot give any more information than the already known
relationship between encoder outputs and true labels. The hope was that
replacing the predicted label $y$ with the true label $y$ would allow for a
final gain in accuracy, but that was not the case in our experiments. There is
a difficult tension to balance: When trusting the predicted $y$ label too
much, the decoder will not be able to perform well on the training dataset
because the predicted label is often wrong. But when we do not trust the
label, we cannot do well at evaluation time.

The toy example indicates that the only other way to gain new information
would be relating the statistical properties of the distribution. However, it
is not clear how to learn this information or how helpful it would be.
Therefore, using label information for a separate dataset appears very
unlikely to improve performance.

A counterpoint to this argument is the performance gap between our model and
the no label input model. This is especially clear in the early stages of
GPT-2 training. In Figure 4, we plot the three accuracies when training GPT-2.
Here, using the ground truth labels for the y dataset gives a better score on
the z dataset than using the model predictions for y. Thus, it seems that we
can conclude that it is possible for the model to learn the joint distribution
$P(Y,Z)$ and use that information effectively. However, the problem is that
the only information about $P(Y,Z)$ that the model is capable of learning is
what can be deduced from $P(X,Y)$ and $P(X,Z)$. In trying to predict $Z$, it
can really only use the information that was learned from $P(X,Z)$, which is
already contained in any normal supervised model. The fact that the model
performance never surpasses the supervised model is evidence toward the
argument that the replacement policy will not help to improve model
performance in general.

##  7 Conclusion

The task of consolidating datasets with different labels and no shared
examples is a hard problem. The experiments did not provide any improvement
over the baseline of only using the $x$ variables. This was surprising, as the
correlation between the $y$ and $z$ labels is quite large. However, this may
be due to an intrinsic difficulty with the setting (no shared examples) rather
than the model design.

##  8 Future Work

Future work of consolidating datasets without shared examples should focus on
using semi-supervised learning with other $x$ labels or supporting the other
dataset labels with metadata.

Another possible direction would be to use the architecture in this paper
together with a subset of shared examples between the datasets. Our approach
can be easily modified to deal with labels that are sometimes missing instead
of all the time. Such a modification could shine in (potentially multi-label)
environments with that require frequent missing value imputation.

## References

  * Caruana (1998) Rich Caruana. 1998.  Multitask learning.  In Sebastian Thrun and Lorien Y. Pratt, editors, _Learning to Learn_ , pages 95–133. Springer.
  * Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In _Proceedings of the 25th international conference on Machine learning_ , pages 160–167.
  * Cordts et al. (2016) Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016.  The cityscapes dataset for semantic urban scene understanding.
  * Das and Petrov (2011) Dipanjan Das and Slav Petrov. 2011.  Unsupervised part-of-speech tagging with bilingual graph-based projections.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL ’11)_ , page Best Paper Award.
  * Daumé (2009) Hal Daumé. 2009.  Frustratingly easy domain adaptation.
  * Kevin Gimpel (2011) Brendan O’Connor Kevin Gimpel, Nathan Schneider. 2011.  Part-of-speech tagging for twitter: Annotation, features, and experiments.  In _Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments_.
  * Nguyen (2020) Dat Quoc Nguyen. 2020.  Bertweet the first large-scale pretrained language model for english tweets.  _VinAi_ , 1.
  * Petrov et al. (2011) Slav Petrov, Dipanjan Das, and Ryan T. McDonald. 2011.  A universal part-of-speech tagset.  _CoRR_ , abs/1104.2086.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _CoRR_.
  * Shen (2007) Libin Shen. 2007.  Guided learning for bidirectional sequence classification.
  * Yijia Liu et al. (2018) Yi Zhu Yijia Liu, Wanxiang Che, and Bing Qin. 2018.  Parsing tweets into universal dependencies.  In _Parsing Tweets into Universal Dependencies_.


# A new method to distinguish gravitational-wave signals from detector noise
transients with Gravity Spy

Seraphim Jarov1<EMAIL_ADDRESS>Sarah Thiele2 Siddharth Soni3 Julian
Ding1,4 Jess McIver1 Raymond Ng4 Rikako Hatoya5 Derek Davis6 1 Department of
Physics and Astronomy, University of British Columbia, Vancouver, British
Columbia, V6T1Z4, Canada  2 Department of Astrophysical Sciences, Princeton
University, 4 Ivy Lane, Princeton, NJ 08544, USA  3 LIGO Lab, Massachusetts
Institute of Technology, Cambridge, MA 02139, USA  4 Department of Computer
Science, University of British Columbia, Vancouver, British Columbia, V6T1Z4,
Canada  5 University of California, Los Angeles Electrical and Computer
Engineering Dept., Los Angeles, CA 90095  6 LIGO, California Institute of
Technology, Pasadena, CA 91125, USA

###### Abstract

The Advanced LIGO and Advanced Virgo detectors have enabled the confident
detection of dozens of mergers of black holes and neutron stars. However, the
presence of detector noise transients (glitches) hinders the search for these
gravitational wave (GW) signals. We prototyped a restructuring of Gravity
Spy’s classification model to distinguish between glitches and astrophysical
signals. Our method is able to correctly classify three-quarters of retracted
candidate events in O3b as non-astrophysical and 100% of the confirmed
astrophysical events as true signals. This approach will inform candidate
event validation efforts in the latest observing run.

††preprint: APS/123-QED

##  I Introduction

Since the first observing run (O1) in 2015, the Advanced LIGO and Advanced
Virgo detectors have seen significant improvements in detector sensitivity [1,
2]. As a result, the rate of confirmed gravitational wave (GW) detections has
increased steadily with each observing run [3, 4, 5]. However, GW detector
data also contains a high rate of non-astrophysical noise transients
(glitches). These glitches come in a variety of morphologies [6, 7, 8, 9, 10].
They often arise at different rates between each detector; and are a result of
many known environmental and instrumental factors as well as unknown sources
[1, 11]. As a result, detector data can become heavily polluted, making
searches for GWs and accurate extraction of source properties substantially
more difficult [1, 12]. Frequent detector glitches can bias analyses when
glitches occur close in time to candidates, or mimic the form of real GW
events and produce false positive candidate events [13]. For example, in the
third observing run (O3), just under a third of open public alerts (OPAs) were
retracted [6, 7, 14]. To aid in event validation and potentially reduce the
rate of retracted event candidates, we propose a quick and accurate method
based on single-detector GW strain data for determining whether a glitch is
present in the data that is robust to loud astrophysical events. This method
will complement existing signal-vs-glitch classification programs, including
the use of Q-occupancy to determine data quality [15], GWSkyNet [16, 17] which
requires multiple detectors, iDQ [18], which requires auxiliary witness data,
and $p_{astro}$ [19] reported in open public alerts [6, 7].

A widely-used machine learning image classifier for GW detector
characterization, Gravity Spy [20, 21, 22], has achieved high accuracies for
classification of detector glitches [13, 23]. The Gravity Spy project
leverages a convolutional neural network (CNN) image classifier using time-
frequency representations of detector data called qscans [20, 24]. Qscans are
also commonly referred to as omega scans [25] and spectrograms [13, 20].

We restructured Gravity Spy’s CNN to provide compact binary coalescence (CBC)
GW signal-vs-glitch classifications which will allow for rapid rejection of
false candidate events using GW strain data. We emphasize that this method
does not require auxiliary witnesses or multiple detectors. Our approach adds
new capability relative to current methods, such as iDQ [18], which requires
auxiliary witness data, or GWSkyNet [16, 17], which requires data from
multiple detectors to classify a candidate event. The purpose of the original
glitch-classification Gravity Spy CNN model [20] was to classify detector
glitches that arise in the LIGO data stream, which it accomplishes with high
accuracy. However, the classifier was not trained to distinguish between
glitches and astrophysical GW signals. In particular, when we first tested our
new model architecture with the original Gravity Spy training set, we found
that simulated GW signals that come from both high mass ($>50M_{\odot}$ total
mass) and low mass ($<50M_{\odot}$ total mass) mergers tend to be
misclassified as glitches that appear similar in appearance, duration, and
frequency range [26]. Examples of these types of glitches and simulated
signals are presented in Figures 1 and 2. This is particularly problematic for
CBC sources with total mass below $30M_{\odot}$, as long duration ($>0.5$
seconds) glitches such as light scattering that mimic these signals have been
common in GW detector data during previous observing runs [4, 6, 23, 27, 28,
29]. We therefore aimed to expand Gravity Spy beyond its current capabilities
by improving signal-vs-glitch classification on top of its glitch
classification framework. We also tackled the challenge of future-proofing our
classification network by considering sources with masses higher than previous
detections [30], as the expected sensitivity improvements [1, 2] in future
observing runs could result in signals from a broader mass spectrum being
detected.

To match the accuracy of Gravity Spy’s glitch classification accuracy for the
signal-vs-glitch case, we introduced a new structure to Gravity Spy’s CNN to
automate the process described in [31]. Motivated by commonalities in glitch
types that are known to mimic the appearance of GW signals from certain mass
ranges, we split the Gravity Spy CNN into two classifiers, one focusing on
distinguishing long duration (i.e. low mass) CBC signals from glitches with
similar time-frequency morphology, and one focusing on short duration (i.e.
high mass) CBC signals and similar glitches. We trained the low mass
classifier on simulated GW signals with total mass between $3M_{\odot}$ and
$50M_{\odot}$ and the high mass classifier on simulated GW signals with total
mass between $50M_{\odot}$ and $250M_{\odot}$. We trained the low mass and
high mass classifiers to be robust against shifts in time-frequency image
centering by augmenting training set images with time offets. Other qscan
processing techniques have also been explored resulting in a third classifier
constructed to classify signals in the $250M_{\odot}$ to $350M_{\odot}$ mass
range. We present a proof-of-principle GW candidate event validation tool that
performs reliably and efficiently.

##  II Methods

###  II.1 Generating Simulated GWs

We first identified a $64$ second segment of quiet LIGO Livingston detector
data111We note that this short data segment is sufficiently representative of
nominal data for this proof-of-principle study. to act as baseline noise we
can inject a simulated waveform into using the PyCBC package [33]. For the
purpose of our study, we simulated signals drawing parameters from a total
mass range of $3M_{\odot}$ to $350M_{\odot}$, a signal-to-noise-ratio (SNR)
range of $3$ to $35$, and a range of $-0.95$ to $0.95$ for both component
spins to span the parameter space of most likely GW sources for the expected
O4 observing run. We injected each generated waveform into the quiet
timeseries data, then used Gravity Spy to process the data as a series of four
qscans, as described in [20]. This workflow can supply qscans of GWs with a
variety of parameters for mass, spin, and SNR. We used this to test Gravity
Spy and our prototype image classifiers leveraging time-frequency
representations of GW detector data.

###  II.2 Generating new training sets

In our study, we constructed and trained two main classifiers which focus on
low-mass and high-mass CBC signals and the types of glitches that have similar
time-frequency morphology in a qscan [13, 20, 23, 25, 34], as outlined in
Table 1. To determine which glitch classes have similar time-frequency
morphology, we first ran the original Gravity Spy on a range of simulated GW
signals, as described in Section IV.1, and included glitch classes where the
original Gravity Spy model confused simulated signals in each mass range for
these classes.

Low mass classifier classes | High mass classifier classes
---|---
Blip | Blip
Low frequency blip | Low frequency blip
Scratchy | Koi Fish
No Glitch | Tomte
GW ($3-50\ M_{\odot}$ total mass CBC) | GW ($50-300\ M_{\odot}$ total mass CBC)
Table 1: The signal and glitch types we used to retrain the Gravity Spy CNN
for optimal signal versus glitch classification, for each classifier.

Equipped with our GW simulation workflow, we retrained Gravity Spy’s CNN (we
use the same CNN as presented in [35]) on enriched training sets that
increased representation of simulated GW signals outside the mass and SNR
ranges present in the original training set. Our enriched training set
contains an equal representation of each class, with roughly 750 examples per
classifier, as equal representation of classes been shown to increase the
robustness of a training set [36]. We discuss the classes included in our
method in the next section.

##  III Challenging types of glitches for different CBC mass ranges

Here we consider particularly challenging cases for a signal-vs-glitch
classifier intaking qscan time series data, especially cases where the time-
frequency morphology of glitches is similar to low mass or high mass CBCs.

###  III.1 Low mass CBCs

CBC sources with a total mass below $50M_{\odot}$ are longer in duration and
generally manifest as lower in energy in a qscan relative to higher mass
(short duration) signals. We present examples in Figures 1 I, II, and III.

|
---
I
|
---
II
|
---
III
|
---
A
|
---
B
|
---
C
Figure 1: Example qscans of low mass CBC GW signals and glitches confused by
the original Gravity Spy model. A $5M_{\odot}$ total mass GW source (I) is
often misclassified as a scratchy glitch (A), which has a similar frequency
range and is also relatively long in duration. A GW source of total mass
$16M_{\odot}$ (II) is often misclassified as ‘no glitch’ (B). This is expected
behavior; there is little difference in a qscan between a low SNR GW signal
and Gaussian detector noise. Lastly, a GW source of $46.5M_{\odot}$ (III) is
commonly confused for a blip glitch (C) due to similar time-frequency
morphology in a qscan.

The original Gravity Spy training set contained very few low mass GW examples.
As a result, the original Gravity Spy model tends to confuse simulated low
mass CBCs with glitches that are also long in duration and have similar
frequency content. The original Gravity Spy model also tends to classify low
mass CBCs, especially simulated GWs with low SNR, as ‘no glitch’. An example
is shown in Figure 1 (II and B). However, we note that this is not an
undesirable outcome. A ‘no glitch’ classification is equivalent to a GW
classification for the purposes of event validation; neither class is
actionable for further data quality investigation.

###  III.2 High mass CBCs

High mass CBCs are shorter in duration in the sensitive band of the LIGO and
Virgo detectors relative to low mass CBCs, as shown in Figure 2 for a
simulated GW sources with total mass $245~{}M_{\odot}$ (I) and
$126~{}M_{\odot}$ (II). These GWs often resemble short duration glitch types
shown in Figure 2, including blips (A) and low frequency blips (B). The
similar frequency range and morphology make distinguishing between high mass
CBCs and these glitch classes difficult. However, high mass CBCs are confused
less often for blip glitches compared to low frequency blips, in part due to
excess power above $300~{}\textrm{Hz}$, which high mass GW sources typically
do not share (see Figure 2).

|
---
I
|
---
II
|
---
D
|
---
A
|
---
B
|
---
C
Figure 2: Example qscans of high mass CBC GW signals and glitches confused by
the original Gravity Spy model. A simulated GW source with a total mass of
$245~{}M_{\odot}$ (I) is most commonly mistaken for blips (A) and low
frequency blips (B), which share a similar morphology in a qscan. In contrast,
a simulated signal with $126~{}M_{\odot}$ total mass and an SNR of 35 (II) is
more often confused for a koi fish glitch (C) or tomte (D) glitch. While the
morphology of these glitches is not as similar, the original Gravity Spy model
tended to confuse higher SNR GW for the higher SNR glitch classes.

For the original Gravity Spy model, high mass GW sources were often confused
for low frequency blips, and sometimes (in $\sim 3\%$ of cases) blip glitches.
This is our main challenge in terms of building a reliable classification
system. In addition, increased GW SNR also led to misclassifications of the
higher SNR glitch classes, koi fish (C) and tomtes (D), likely due to the
limited representation of louder GW signals in the original Gravity Spy
training set. Our investigation into louder GW signals found that $\sim 50\%$
of misclassifications of high mass CBCs were classified as koi fish or tomte
glitches. As a result, we incorporated higher SNR signals into the training
set of our new signal-vs-glitch classifiers.

###  III.3 Extremely high mass CBCs

For mergers beyond $250M_{\odot}$, the original Gravity Spy model classifies
$100\%$ of these signals as low frequency blips with $\sim 99\%$ confidence.
For example, a $270M_{\odot}$ CBC signal shares strong similarities in
duration, frequency range, and morphology with low frequency blips, as shown
in Figure 3. Since the original Gravity Spy model could not classify any CBCs
above $250M_{\odot}$, we investigated these types of signals separately and
labeled them as “extremely high mass” GW sources. Further investigation,
described in Section IV.4, suggests that treating this class separately is
necessary.

Figure 3: On the left is a simulated CBC GW signal with a total mass of
$281~{}M_{\odot}$ and on the right is an example of a low frequency blip
glitch. Signals with mass above $250~{}M_{\odot}$ overlap very closely with
low frequency blip glitches in frequency range and duration, which makes them
particularly difficult for image classifiers to differentiate.

##  IV Gravity Spy as a base for a signal-vs-glitch classifier

We first describe our investigations of the original Gravity Spy model’s
potential to distinguish between GW signals and detector glitches. We then
discuss our improvements to the original Gravity Spy model for use in this new
capacity, targeting the challenges of the classes discussed in the previous
section.

###  IV.1 Testing the original Gravity Spy

To understand the performance of the original Gravity Spy model as a possible
signal-vs-glitch classifier, we investigated the model’s classification of
simulated GWs as a function of the total mass, SNR, and spin of a CBC signal.
We generated a set of simulated signals and tested whether the model would
classify them as chirps; the class label corresponding to the limited number
of simulated GWs included in Gravity Spy’s original training set.

The parameters for the simulated GW waveforms used in the test set were drawn
uniformly from intervals: $\text{mass}_{i}\in[1M_{\odot},125M_{\odot}]$,
$\text{SNR}\in[3,25]$, and $\text{spin}_{i}\in[-0.95,0.95]$, where
$i\in\\{1,2\\}$ corresponds to each compact object in the simulated merger. We
split the simulated signals into the low mass ($3-50~{}M_{\odot}$) and high
mass ($50-250~{}M_{\odot}$) categories such that there were 500 examples in
each mass range.

Figure 4 shows the original Gravity Spy’s classification222Note that while
Gravity Spy produces confidence ratings for each class a GW signal or glitch
could be classified as, our investigation was only interested in Gravity Spy’s
ability to differentiate between GWs and glitches. So, in our analysis of
Gravity Spy classifications, only the classification with the highest
confidence was considered. of these simulated signals, plotted by the
component masses $\text{mass}_{i}$, with classification accuracy shown as
color. Of the $273$ false negatives in the low mass range (Figure 4 A), $81\%$
were classified as a scratchy glitch while $17\%$ were classified as Gaussian
noise (‘no glitch’). We note again that for the purposes of this study, a no
glitch classification is equivalent to a GW classification, as both classes
are not actionable for further data quality studies in GW event candidate
validation. The misclassifications of higher mass simulated GW signals (Figure
4 B) were slightly more diverse with the majority of the $204$ false negatives
being classified as either a blip or low frequency blip.

|
---
A
|
---
B
Figure 4: The original Gravity Spy model’s classification of $500$ simulated
signals. The left plot (A) shows low mass sources with parameters
$\text{mass}_{i}\in[1M_{\odot},25M_{\odot}]$ where $i\in\\{1,2\\}$ corresponds
to each binary component in the simulated merger, $\text{SNR}\in[3,25]$, and
$\text{spin}_{i}\in[-0.95,0.95]$. The right plot (B) shows high mass sources
with parameters $\text{mass}_{i}\in[25M_{\odot},125M_{\odot}]$ with the same
spin and SNR ranges as in (A). ”Chirps” that have been correctly classified by
the original Gravity Spy model are colored yellow, and incorrect
classifications are purple. While the simulated GWs have a range of SNR and
spin values, the regions where the original Gravity Spy model is able to
perform best are clearly dominated by the source’s total mass, motivating a
method that targets the corresponding morphological differences.

These results are evidence that using the original Gravity Spy’s CNN outside
of its original intended purpose is not sufficient for an accurate signal-vs-
glitch classifier without further modification, as we require high accuracy
for automation for GW candidate event validation. In particular, GWs require
more representation in our training set relative to the original Gravity Spy
training set. Our results are consistent with the findings reported in
Bahaadini et al. [21], which identified a higher rate of inaccuracies
associated with signals that had a poorer representation in the original
Gravity Spy training set.

Only 60 out of the 9631 original Gravity Spy training set examples were
simulated GWs, as shown in Table 2. These simulated GWs were labeled as
“chirps” to also capture potential transient noise increasing in frequency
over time with a similar morphology [20]. Given the lack of GW examples, we
might expect few correct classifications of a broad range of simulated GWs as
“chirps”. However $79\%$ of GW sources with a total mass within $25M_{\odot}$
and $150M_{\odot}$ were correctly classified by the original Gravity Spy
model.

Glitch |  Training Examples |  Glitch |  Training Examples
---|---|---|---
Blip |  1821 |  Koi Fish |  706
Tomte |  703 |  LF Blip |  630
LF Burst |  621 |  Scattered Light |  593
Light Modulation |  512 |  Power Line |  449
LF Lines |  447 |  Extremely Loud |  447
Violin Mode |  412 |  Fast Scattering |  400
Scratchy |  337 |  1080 Lines |  327
Whistle |  299 |  Helix |  279
Repeating Blips |  263 |  No Glitch |  117
1400 Ripples |  81 |  Chirp |  60
Air Compressor |  58 |  Wandering line |  42
Paired Doves |  279 |  |
Table 2: Gravity Spy’s original training set [22]. Of the 9631 training
examples in the original training set, the number of simulated GW (“chirp”)
examples was particularly small, as GW classification was outside of Gravity
Spy’s original intended use.

In summary, we identified a clear relation between total mass and the original
Gravity Spy model’s classification accuracy of GWs, motivating splitting up
the classifier to target different mass ranges. Additionally, we showed that
only a small subset of glitch classes cause GW misclassifications in the
original Gravity Spy model, which is evidence that 20 glitch classes are
unnecessary for reliable signal versus glitch classification and may be
introducing confusion for this targeted application. These results suggest
that better results for signal-vs-glitch classification are attainable with an
augmented training set and a restructuring of the Gravity Spy classification
model.

###  IV.2 Rebuilding Gravity Spy

We propose a restructuring of the original Gravity Spy classification
framework for use in signal-vs-glitch classification. Our proposed model
consists of two classifiers focused on signals from either end of the mass
spectrum and the corresponding glitch classes that we have found the original
Gravity Spy model confuses for signals.

We will call the signal versus glitch model that focuses on low mass CBC
sources and similar glitches GSpySVG_LM and the one that focuses on high mass
CBCs and similar glitches GSpySVG_HM. In our prototype of this method we
supply a balanced training set for each model, reduce the glitch classes to
only those that the original Gravity Spy model confused with signals in the
corresponding mass range, and increased the diversity of GW morphology
included in the GW class to better represent expected GW signals.

GSpySVG_LM

|  | # of Examples
---|---|---
| Blip |  |  | 150
| LF Blip |  |  | 150
| No Glitch |  |  | 115
| Scratchy |  |  | 150
True Class | LM GW |  |  | 150

GSpySVG_HM

|  | # of Examples
---|---|---
| Blip |  |  | 150
| LF Blip |  |  | 150
| Koi Fish |  |  | 150
| Tomte |  |  | 150
True Class | HM GW |  |  | 150

Table 3: Our new method consists of two classifiers based on the Gravity Spy
model. GSpySVG_LM is trained to distinguish between low mass signals and
similar glitches and GSpySVG_HM is trained to handle high mass signals and
similar glitches. We list the glitch classes used in each classifier as well
as the number of examples used in the corresponding training set. Although
this prototype method uses fewer training examples, it has a more equal
representation of GWs relative to glitch classes compared to the original
Gravity Spy training set. We only included glitch classes that Gravity Spy’s
original model confused with either low mass or high mass GW signals.

After we implemented these changes, we found results significantly improved
compared with the original Gravity Spy model. Using the same tests sets and
mass ranges as reported for the original Gravity Spy model in section IV.1
Figure 4, we evaluated our new method’s ability to correctly classify
simulated GW signals, as shown in Figure 5. Most notably, the number of
simulated signals in the $3-250M_{\odot}$ range classified correctly is much
higher for GSpySVG_HM and GSpySVG_LM compared with the performance of original
Gravity Spy, as shown in Figures 4 and 5 respectively. While the original
Gravity Spy model was only able to classify $52\%$ of simulated signals in the
same mass range correctly, GSpySVG_HM and GSpySVG_LM were able to classify
$97\%$ of them correctly.

|
---
A
|
---
B
Figure 5: GSpySVG_LM (A) and GSpySVG_HM (B) tested on the same test set and
mass range used for the original Gravity Spy model, as reported in Figure 4.
As in Figure 4, the data is presented with the mass values on the $x$ and $y$
axes, while signals that have been correctly identified are colored yellow and
incorrect classifications are purple. Compared with the original Gravity Spy
model, our prototype produces a substantial decrease in GW misclassifications:
273 to 9 for simulated signals with total mass in the range $3-50M_{\odot}$
and 204 to 19 for simulated signals with total mass in the range
$50-300M_{\odot}$.

We also observed similarly accurate results for glitch classifications by
GSpySVG_HM and GSpySVG_LM. We found that both GSpySVG_HM and GSpySVG_LM were
able to correctly classify $99\%$ of the $200$ test glitches as belonging to
the correct glitch class. This shows our method is able to increase signal
classification accuracy while maintaining strong performance in glitch
classification. We note, however, that novel glitch classes that could appear
in future observing runs were not considered as the focus of this proof-of-
principle study was to demonstrate feasibility. For a study on the robustness
of a wider range of glitch classes we refer the reader to the follow-up paper
by Alvarez et al [38].

GSpySVG_LM Classification

|  |  Blip |  LF Blip |  No Glitch |  Scratchy |  GW
---|---|---|---|---|---|---
| Blip | 50 | 0 | 0 | 0 | 0
| LF Blip | 0 | 50 | 0 | 0 | 0
| No Glitch | 0 | 0 | 49 | 1 | 0
| Scratchy | 0 | 0 | 0 | 48 | 2
True Class | GW | 0 | 0 | 8 | 1 | 491

GSpySVG_HM Classification

|  |  Blip |  LF Blip |  Koi Fish |  Tomte |  GW
---|---|---|---|---|---|---
| Blip | 50 | 0 | 0 | 0 | 0
| LF Blip | 0 | 50 | 0 | 0 | 0
| Koi Fish | 0 | 0 | 47 | 3 | 0
| Tomte | 0 | 0 | 0 | 50 | 0
True Class | GW | 4 | 15 | 0 | 0 | 481

Table 4: A confusion matrix for GSpySVG_LM (left) and GSpySVG_HM (right)
performance, as shown in Figure 5. Both GSpySVG_LM and GSpySVG_HM correctly
classified $99\%$ of the test glitches while GW classification accuracy was
98$\%$ and 96$\%$ respectively. Moreover, below a total mass of
$200M_{\odot}$, GSpySVG_HM was able to classify $98\%$ of simulated signals
correctly.

Further testing showed GSpySVG_LM and GSpySVG_HM correctly classified $75\%$
of the retracted O3b GW event candidates [6, 7] as non-astrophysical.
Additionally, we tested the models on all confirmed O3b candidate events [6].
GSpySVG_HM was able to correctly classify all 10 signals that fell under the
high mass GW class and GSpySVG_LM correctly classified 9/11. The two O3b
confirmed GW candidates misclassified by GSpySVG_LM were both classified as no
glitch. So, of all 21 signals in O3b, none would have been flagged as a glitch
by these new models.

###  IV.3 Offsets

We also considered glitches and GW signals with a slight time offset relative
to the reported merger time of the candidate. We confirmed that, as previously
reported, shifting signals with respect to the center of the qscan results in
a significant decrease in Gravity Spy classification performance. In
particular, we saw a $16\%$ decrease in signal classification accuracy when we
introduced random offsets in the range of $[-0.1,0.1]$ seconds to account for
possible offset candidate event times reported in the search pipelines.

To improve robustness against time offsets for GW event candidates, we
supplemented our new training set for the GSpySVG_LM and GSpySVG_HM models
with time-translated examples. We made $4$ copies of each example in our
GSpySVG_LM and GSpySVG_HM training sets and added a random offset in the range
of $[-0.1,0.1]$ seconds to each of the copies. This supplemented training set
increased classification accuracy from $83\%$ to $99\%$ on the same test set
of time-translated images. We saw a negligible difference in performance on
the un-translated test set. Figure 6 gives an example of a simulated GW signal
in our original training set and $4$ copies with random time offsets, as used
in our supplemented training set.

Figure 6: An example of a simulated signal with random time offsets. A
simulated GW with a merger time at $t=0$ is shown on the top left. The other
images are $4$ copies, each containing a random time offset in the range of
$[-0.1,0.1]$ seconds. We added red center lines at $t=0$ to the images
included in this figure as a visual aid.

###  IV.4 Re-scaled qscans for CBC sources with total mass above
$250M_{\odot}$

In section III.3, we noted that signals with a total mass above $250M_{\odot}$
were extremely challenging for the original Gravity Spy CNN to classify. When
testing the original Gravity Spy model on a test set of 50 simulated signals
with total masses between $250M_{\odot}$ and $350M_{\odot}$, none of them were
classified correctly. Even after retraining for signals in the $50$ to
$250M_{\odot}$ range, GSpySVG_HM was only able to correctly classify $\sim
8\%$ of these signals. Both the original Gravity Spy model and GSpySVG_HM
misclassified these simulated signals as low frequency blip glitches.

In this mass range, signals and low frequency blip glitches are nearly
indistinguishable by eye and with a CNN using qscans as a feature set, as
shown in Figure 3. To overcome this challenge, we introduce a new feature set
using qscan processing techniques that give our method more distinguishing
power.

We apply a Mercator projection to the qscans that stretches images vertically
[39]; an example of a simulated GW source with a total mass of
$270~{}M_{\odot}$ and low frequency blip before and after applying the
projection can be seen in Figure 7. This projection gives us better resolution
of the data above $\sim 100$ Hz. As shown in Figure 7, the GW (A) presents as
a long thick line whereas the low frequency blip (D) tapers off and becomes
quite thin. Through the lens of the Mercator projection, these characteristics
are common between low frequency blips and GW sources in the
$[250M_{\odot},350M_{\odot}]$ range.

|
---
A
|
---
B
|
---
C
|
---
D
Figure 7: An example of a simulated $270M_{\odot}$ merger (A) and the same
signal after applying the Mercator projection (B). We show the same projection
for a low frequency blip glitch in plots C and D. We note the characteristic
thinning of the low frequency blip glitch close to the bottom of plot D. One
technical note is that the Mercator transform does alter the frequency labels
that are present in the qscans above, however, they can be ignored as Gravity
Spy does not read labels.

We retrained another classifier using Gravity Spy’s architecture on a training
set of simulated signals in the total mass range $[250M_{\odot},350M_{\odot}]$
and low frequency blip examples with the Mercator transformation applied to
the qscans. We call this binary classification model GSpySVG_EHM. On the same
test set that the original Gravity Spy model classified $0/50$ extremely high
mass simulated signals correctly, GSpySVG_EHM was able to correctly classify
$48/50$. Moreover, our binary model saw a $100\%$ correct classification rate
on $50$ test low frequency blip glitches. GSpySVG_EHM acts as a proof of
concept that building different feature sets with qscan processing techniques
can aid in challenging classifications. A first implementation based on the
recommendations from this proof-of-principle study leveraged this key result
by funneling a candidate’s classification to a low-mass or high-mass
classifier based on its initial estimated mass [38].

##  V Summary and conclusion

In this paper, we have discussed the types of GW detector noise artifacts that
have been known to mimic real GW events and have outlined our investigation
into prototyping a reliable signal-vs-glitch classification model using
Gravity Spy’s architecture.

The main finding of this paper is that breaking up the original Gravity Spy
model into multiple classifiers that specialize in differentiating similar CBC
signals and detector glitches results in significant improvements for signal-
vs-glitch classification accuracy. We have prototyped a restructuring of the
original Gravity Spy model in the GSpySVG_LM and GSpySVG_HM models which have
shown great improvements when tested on simulated signals, glitches, and O3b
candidate events.

We compared the performance of our method to using the original Gravity Spy
model as a signal-vs-glitch classifier, outside of its designed scope of use.
In their target mass ranges, GSpySVG_LM and GSpySVG_HM achieved an overall
simulated signal classification accuracy of $97\%$, an increase from $52\%$
with the original Gravity Spy model. Furthermore, GSpySVG_LM and GSpySVG_HM
classified test sets consisting of 200 glitch examples that often mimic the
form of signals with $99\%$ accuracy each. Both GSpySVG_LM and GSpySVG_HM were
also tested on O3b data where they were able to correctly classify all
confirmed events as non-glitches, an improvement from $6/21$ with the original
Gravity Spy model. In addition to our main improvements in the $<250M_{\odot}$
range, we also investigated signals and glitches that may appear with a slight
offset in time. For signals with a time offset in the range of $\pm 0.1$
seconds, we were able to increase classification accuracy from $83\%$ to
$99\%$ by adding random time offsets to each of the training images in the
GSpySVG_LM and GSpySVG_HM training sets.

We further improved the performance of our prototype method to high mass CBC
sources that share morphology with low frequency blip glitches. We developed
novel qscan scaling techniques and demonstrated that Gravity Spy’s CNN is able
to distinguish between signals with total mass greater than $250M_{\odot}$ and
low frequency blip glitches when a Mercator projection is applied to the
qscans. We trained a third binary classifier that made use of the Mercator
projection to emphasize distinguishing features between the extremely high
mass GW and low frequency blip classes. This GSpySVG_EHM model classified
simulated signals with a total mass between $250M_{\odot}$ and $350M_{\odot}$
with $96\%$ accuracy. This is a substantial improvement over $0\%$ accuracy
that we saw with the original Gravity Spy model. This part of the project
highlights the potential for qscan processing techniques to be leveraged for
CNN classifications.

We note that, at the time of writing, a complete workflow based on this work
that combines the GSpySVG_LM, GSpySVG_HM, and GSpySVG_EHM models into a
reliable signal-vs-glitch classifier for future GW observing runs has been
implemented by Alvarez et al. [38].

Acknowledgements: SJ was supported by the NSERC USRA program. SS acknowledges
support from the United States National Science Foundation (NSF) under award
PHY-1764464 to the LIGO Laboratory and NSF grant PHY-1806656. JM was supported
by the Canada Research Chairs programs. DD is supported by the NSF as a part
of the LIGO Laboratory. LIGO was constructed by the California Institute of
Technology and Massachusetts Institute of Technology with funding from the
National Science Foundation, and operates under cooperative agreement
PHY-1764464. This material is based upon work supported by NSF’s LIGO
Laboratory which is a major facility fully funded by the National Science
Foundation. The authors are grateful for computational resources provided by
the LIGO Laboratory and supported by National Science Foundation Grants
PHY-0757058 and PHY-0823459.

## References

  * [1] Derek Davis, Joseph S Areeda, Beverly K Berger, et al.  LIGO detector characterization in the second and third observing runs.  Classical and Quantum Gravity, Apr 2021.
  * [2] B. P. Abbott, R. Abbott, T. D. Abbott, et al.  Low-latency gravitational-wave alerts for multimessenger astronomy during the Second Advanced LIGO and Virgo observing run.  The Astrophysical Journal, 875(2):161, apr 2019.
  * [3] J Aasi, B P Abbott, R Abbott, et al.  Advanced LIGO.  Classical and Quantum Gravity, 32(7):074001, Mar 2015.
  * [4] F Acernese, M Agathos, K Agatsuma, et al.  Advanced Virgo: a second-generation interferometric gravitational wave detector.  Classical and Quantum Gravity, 32(2):024001, Dec 2014.
  * [5] Jess McIver and D. H. Shoemaker.  Discovering gravitational waves with advanced LIGO.  Contemporary Physics, 61(4):229–255, 2020.
  * [6] The LIGO Scientific Collaboration, the Virgo Collaboration, the KAGRA Collaboration, et al.  GWTC-3: Compact binary coalescences observed by LIGO and Virgo during the second part of the third observing run, 2021.
  * [7] The LIGO Scientific Collaboration, the Virgo Collaboration, R. Abbott, et al.  GWTC-2.1: Deep extended catalog of compact binary coalescences observed by LIGO and Virgo during the first half of the third observing run, 2021.
  * [8] B P Abbott, R Abbott, T D Abbott, et al.  Characterization of transient noise in advanced LIGO relevant to gravitational wave signal GW150914.  33(13):134001, jun 2016.
  * [9] L Blackburn, L Cadonati, S Caride, et al.  The LSC glitch group: monitoring noise transients during the fifth LIGO science run.  Classical and Quantum Gravity, 25(18):184004, Sep 2008.
  * [10] S. Coughlin, S. Bahaadini, N. Rohani, et al.  Classifying the unknown: Discovering novel gravitational-wave detector glitches using similarity learning.  Physical Review D, 99(8), Apr 2019.
  * [11] Beverly K. Berger.  Identification and mitigation of advanced LIGO noise sources.  Journal of Physics: Conference Series, 957:012004, feb 2018.
  * [12] B P Abbott, R Abbott, T D Abbott, et al.  Effects of data quality vetoes on a search for compact binary coalescences in Advanced LIGO’s first observing run.  Classical and Quantum Gravity, 35(6):065010, feb 2018.
  * [13] Derek Davis, Laurel V White, and Peter R Saulson.  Utilizing aLIGO glitch classifications to validate gravitational-wave candidates.  Classical and Quantum Gravity, 37(14):145001, Jun 2020.
  * [14] The LIGO Scientific Collaboration and The Virgo Collaboration.  ”LIGO/Virgo/Kagra Public Alerts user guide”.  https://emfollow.docs.ligo.org/userguide/, 2022.
  * [15] Siddharth Soni, Ethan Marx, Erik Katsavounidis, et al.  QoQ: a Q-transform based test for Gravitational Wave transient events.  5 2023.
  * [16] Miriam Cabero, Ashish Mahabal, and Jess McIver.  Gwskynet: A real-time classifier for public gravitational-wave candidates.  The Astrophysical Journal Letters, 904(1):L9, nov 2020.
  * [17] Thomas C. Abbott, Eitan Buffaz, Nicholas Vieira, et al.  GWSkyNet-multi: A machine-learning multiclass classifier for LIGO–virgo public alerts.  The Astrophysical Journal, 927(2):232, mar 2022.
  * [18] Reed Essick, Patrick Godwin, Chad Hanna, et al.  iDQ: Statistical inference of non-gaussian noise with auxiliary degrees of freedom in gravitational-wave detectors.  Machine Learning: Science and Technology, 2(1):015004, dec 2020.
  * [19] Shasvath J Kapadia, Sarah Caudill, Jolien D E Creighton, et al.  A self-consistent method to estimate the rate of compact binary coalescences with a poisson mixture model.  Classical and Quantum Gravity, 37(4):045007, jan 2020.
  * [20] M Zevin, S Coughlin, S Bahaadini, et al.  Gravity spy: integrating advanced LIGO detector characterization, machine learning, and citizen science.  Classical and Quantum Gravity, 34(6):064003, Feb 2017.
  * [21] S. Bahaadini, V. Noroozi, N. Rohani, et al.  Machine learning for gravity spy: Glitch classification and dataset.  Information Sciences, 444:172–186, 2018.
  * [22] J. Glanzer et al.  Data quality up to the third observing run of advanced LIGO: Gravity Spy glitch classifications.  Class. Quant. Grav., 40(6):065004, 2023.
  * [23] S Soni, C P L Berry, S B Coughlin, et al.  Discovering features in gravitational-wave data through detector characterization, citizen science and machine learning.  Classical and Quantum Gravity, 38(19):195016, sep 2021.
  * [24] Shourov Chatterji.  The search for gravitational wave bursts in data from the second LIGO science run.  2005\.
  * [25] D Davis, J Areeda, B Berger, et al.  LIGO detector characterization in the second and third observing runs.  Classical and Quantum Gravity, 38, 07 2021.
  * [26] S. Thiele, J McIver, E. Goetz, and B. Berger.  Final results from gravity spy safety study.  January 2021 DetChar F2F meeting, 2021.  Accessed: 2021–09-18.
  * [27] S. Soni et al.  Reducing scattered light in LIGO’s third observing run.  Class. Quant. Grav., 38(2):025016, 2020.
  * [28] Jane Glanzer, Siddharth Soni, Jaidyn Spoon, et al.  Noise in the LIGO Livingston Gravitational Wave Observatory due to Trains.  4 2023.
  * [29] D Davis, J S Areeda, B K Berger, et al.  LIGO detector characterization in the second and third observing runs.  Classical and Quantum Gravity, 38(13):135014, jun 2021.
  * [30] B. P. Abbott, R. Abbott, T. D. Abbott, et al.  Prospects for observing and localizing gravitational-wave transients with Advanced LIGO, Advanced Virgo and KAGRA.  Living Reviews in Relativity, 23(1), sep 2020.
  * [31] Derek Davis, Laurel V White, and Peter R Saulson.  Utilizing aLIGO glitch classifications to validate gravitational-wave candidates.  Classical and Quantum Gravity, 37(14):145001, jun 2020.
  * [32] We note that this short data segment is sufficiently representative of nominal data for this proof-of-principle study.
  * [33] Samantha A Usman, Alexander H Nitz, Ian W Harry, et al.  The PyCBC search for gravitational waves from compact binary coalescence.  Classical and Quantum Gravity, 33(21):215004, Oct 2016.
  * [34] Miriam Cabero, Andrew Lundgren, Alex Nitz, et al.  Blip glitches in Advanced LIGO data, 01 2019.
  * [35] M Zevin, S Coughlin, S Bahaadini, et al.  Gravity spy: integrating Advanced LIGO detector characterization, machine learning, and citizen science.  Classical and Quantum Gravity, 34(6):064003, feb 2017.
  * [36] Trevor Hastie, Robert Tibshirani, and Jerome Friedman.  The Elements of Statistical Learning.  Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
  * [37] Note that while Gravity Spy produces confidence ratings for each class a GW signal or glitch could be classified as, our investigation was only interested in Gravity Spy’s ability to differentiate between GWs and glitches. So, in our analysis of Gravity Spy classifications, only the classification with the highest confidence was considered.
  * [38] Sofia Alvarez-Lopez, Annudesh Liyanage, Julian Ding, et al.  Gspynettree: A signal-vs-glitch classifier for gravitational-wave event candidates, 2023.
  * [39] Frederick Pearson.  Map projections: Theory and applications.  1990\.