# Citizen ASAS-SN Data Release I: Variable Star Classification Using Citizen
Science

C. T. Christy,1 T. Jayasinghe,1,2 K. Z. Stanek1,2 C. S. Kochanek1,2 Z. Way,3
J. L. Prieto,4 B. J. Shappee,5 T. W.-S. Holoien,6 T. A. Thompson,1,2,7 A.
Schneider8
1Department of Astronomy, The Ohio State University, 140 West 18th Avenue,
Columbus, OH 43210, USA
2 Center for Cosmology and Astroparticle Physics, The Ohio State University,
191 W. Woodruff Avenue, Columbus, OH 43210
3Department of Physics and Astronomy, Georgia State University, Atlanta GA
30303, USA
4 Núcleo de Astronomía de la Facultad de Ingeniería y Ciencias, Universidad
Diego Portales, Av. Ejército 441, Santiago, Chile
5 Institute for Astronomy, University of Hawai’i, 2680 Woodlawn Drive,
Honolulu, HI 96822,USA
6 The Observatories of the Carnegie Institution for Science, 813 Santa Barbara
Street, Pasadena, CA 91101, USA
7 Millennium Institute of Astrophysics, Santiago, Chile
8ASC Technology Services, 433 Mendenhall Laboratory 125 South Oval Mall
Columbus OH, 43210, USA
E-mail<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

We present the first results from Citizen ASAS-SN, a citizen science project
for the All-Sky Automated Survey for Supernovae (ASAS-SN) hosted on the
Zooniverse platform. Citizen ASAS-SN utilizes the newer, deeper, higher
cadence ASAS-SN $g$-band data and tasks volunteers to classify periodic
variable star candidates based on their phased light curves. We started from
40,640 new variable candidates from an input list of ${\sim}7.4$ million stars
with $\delta<-60^{\circ}$ and the volunteers identified 10,420 new discoveries
which they classified as 4,234 pulsating variables, 3,132 rotational
variables, 2,923 eclipsing binaries, and 131 variables flagged as Unknown.
They classified known variable stars with an accuracy of 89% for pulsating
variables, 81% for eclipsing binaries, and 49% for rotational variables. We
examine user performance, agreement between users, and compare the citizen
science classifications with our machine learning classifier updated for the
$g$-band light curves. In general, user activity correlates with higher
classification accuracy and higher user agreement. We used the user’s "Junk"
classifications to develop an effective machine learning classifier to
separate real from false variables, and there is a clear path for using this
"Junk" training set to significantly improve our primary machine learning
classifier. We also illustrate the value of _Citizen ASAS-SN_ for identifying
unusual variables with several examples.

###### keywords:

stars:variables – stars:binaries:eclipsing – stars:rotation – Light Curves –
Stellar Classification – catalogues – surveys

††pubyear: 2021††pagerange: Citizen ASAS-SN Data Release I: Variable Star
Classification Using Citizen Science–References

##  1 Introduction

Variable stars are some of the most useful astrophysical tools as they are
used to probe many aspects of stellar evolution and galactic structure.
Eclipsing binaries allow the derivation of empirical calibrations for
fundamental stellar parameters such as mass and radii (Torres et al., 2009).
The period-luminosity relation of Cepheids is crucial to probing cosmological
distances (Leavitt, 1908; Freedman et al., 2019; Riess et al., 2018). The
short period $\delta$ Scuti variables allow us to study the scaling relations
between stellar parameters (effective temperature, surface gravity, density,
etc.) and astroseismology (Hasanzadeh et al., 2021). For researchers to truly
utilize these systems, it is important that they be discovered and classified.

The search for new variable stars is now dominated by large surveys. This
includes surveys such as the All-Sky Automated Survey (ASAS; Pojmanski 2002),
the All-Sky Automated Survey for SuperNovae (ASAS-SN; Shappee et al. 2014;
Kochanek et al. 2017; Jayasinghe et al. 2018, 2021), the Asteroid Terrestrial-
impact Last Alert System (ATLAS; Tonry et al. 2018a; Heinze et al. 2018), the
Catalina Real-Time Transient Survey (CRTS; Drake et al. 2009), EROS (Derue et
al. 2002), Gaia (Prusti et al., 2016; Brown et al., 2018), MACHO Alcock et al.
2000, the Northern Sky Variability Survey (NSVS; Woźniak et al. 2004), the
Optical Gravitational Lensing Experiment (OGLE; Udalski 2004), and the Zwicky
Transient Facility (ZTF; Bellm 2014).

ASAS-SN is a wide-field photometric survey that monitors the entire night sky
using 20 telescopes located in both the Northern and Southern hemispheres
(Shappee et al., 2014; Kochanek et al., 2017; Jayasinghe et al., 2018). ASAS-
SN detects variables and other transients in the process of finding bright
supernovae (Holoien et al., 2016). For the initial V-band catalog of
variables, ${\sim}60$ million stars were classified through machine learning
techniques, resulting in a catalog of ${\sim}426,000$ variables, of which
${\sim}220,000$ were new discoveries (Jayasinghe et al., 2020, 2021).

Using machine learning techniques to identify and classify variable stars is
particularly efficient for common variable classes and other known phenomena.
However, some object classes are ambiguous and noise or systematic errors will
sometimes confuse the classifiers. We can address this problem by using
citizen science to classify variable star candidates in ASAS-SN along with
machine learning. Citizen science may also more effectively identify rare
phenomena compared to a machine learning classifier due to their scarcity in
the training data (Alhammady & Ramamohanarao, 2004). The ASAS-SN citizen
science project, _Citizen ASAS-SN_ , is hosted on the
Zooniverse111Zooniverse:https://www.zooniverse.org/ platform and aims to
assist in the classification of variable stars. The Zooniverse is the worlds
largest hub for citizen science; in recent years, it has hosted many
successful projects that often lead to serendipitous discoveries (Trouille et
al., 2019).

Here we analyze the first results of _Citizen ASAS-SN_. We examine the
classifications made by the citizen scientists and their ability to correctly
label variable stars from their light curves. Through their classifications,
we have discovered 10,420 new variable stars and flagged many interesting
variables for follow-up studies. We find that citizen scientists can reliably
separate “junk” sources from real variable stars and distinguish between
pulsating variables and eclipsing binaries. We also outline our new $g$-band
machine learning classifier and discuss its performance compared to the
citizen scientists. In Section $\S 2$ we describe the ASAS-SN data used to
generate light curves. Section $\S 3$ discusses the new $g$-band machine
learning classifier. We outline the details of _Citizen ASAS-SN_ in Section
$\S 4$, along with an analysis of the classifications made by the citizen
scientists. In Section $\S 5$ we compare the machine learning and citizen
science classifications. We highlight some of the interesting variable stars
our users encountered in Section $\S 6$ and discuss the utility of citizen
science in identifying such systems. We present a summary of our work in
Section $\S 7$.

##  2 The ASAS-SN $g$ \- band catalog of variable stars

Starting in 2014, ASAS-SN began surveying the sky in the $V$-band with a
limiting magnitude of $V\lesssim 17$ mag and a $\sim 2-3$ day cadence using 8
telescopes on two mounts in Chile and Hawaii. Each ASAS-SN camera takes 3
images with 90 second exposures for each epoch. The field of view of an ASAS-
SN camera is 4.5 deg2, the pixel scale is 8$\aas@@fstack{\prime\prime}$0 and
the FWHM is typically ${\sim}2$ pixels. ASAS-SN uses image subtraction (Alard
& Lupton, 1998; Alard, 2000) for the detection of transients and variable
sources. Since 2018, ASAS-SN has shifted to the $g$-band and expanded to 20
cameras on 5 mounts, adding new units in South Africa, Texas, and Chile. All
of the ASAS-SN telescopes are hosted by the Las Cumbres Observatory (LCO;
Brown et al. 2013). When compared to the V-band data, the $g$-band data has an
improved depth ($g\lesssim 18.5$ mag), cadence ($\lesssim 24$ hours in the
$g$-band vs. ${\sim}2-3$ days in the $V$-band), and reduced diurnal aliasing
due to the longitudinal spread of the ASAS-SN units.

As our input source catalog for this project, we used the `refcat2` catalog
(Tonry et al., 2018b). For this paper, we selected all sources with
declinations $\delta<-60^{\circ}$, $g<18$ mag and
$r_{1}<30\aas@@fstack{\prime\prime}0$, where the `refcat2` metric `r1` is the
radius at which the cumulative $G$ flux in the aperture exceeds the flux of
the source being considered and is a measure of blending around a star. After
applying these selection criteria, we were left with ${\sim}7.4$ million
sources. We extracted their $g$-band light curves as described in Jayasinghe
et al. (2018) using image subtraction (Alard & Lupton, 1998; Alard, 2000) and
aperture photometry on the subtracted images with a 2 pixel radius aperture.
We corrected the zero point offsets between the different cameras as described
in Jayasinghe et al. (2018) and calculated periodograms using the Generalized
Lomb-Scargle (GLS, Zechmeister & Kürster 2009; Scargle 1982) algorithm.

Candidate variable sources were identified using various cuts in light curve
(for e.g., median magnitude, root-mean-square deviation, and string length
statistics) and GLS periodogram statistics (power and false alarm probability)
as summarized in Jayasinghe et al. (2019a). The _Citizen ASAS-SN_ workflow
presently focuses on the classification of periodic variable stars, so we did
not include non-periodic sources in this work. We required that the false
alarm probability for the period is better than ${10^{-7}}$ for sources with
median magnitudes fainter than ${g}=16.5$ mag. Figure 1 shows the distribution
of variables by their average $g$-band magnitude. Variable sources with
magnitudes of $g\leq 11.5$ were considered to be saturated. Note the peak at
the faint end of the distribution. It comes from removing some candidate
selection criteria used by Jayasinghe et al. (2018) with consequences we did
not fully appreciate at the time (see §4.4).

The final product of this paper is the first installment of the ASAS-SN
$g$-band catalog of variable stars. This includes classification data from our
updated machine learning classifier as well as the input from our citizen
scientists. We also include supplementary data from crossmatches to existing
photometric catalogs. The revised catalog is available at https://asas-
sn.osu.edu/variables.

Figure 1: Number distribution of the mean $g$-band magnitudes for the
candidate variable sources with $\delta<-60^{\circ}$. Retrospectively, the
peak at faint magnitudes mainly consists of false positive candidates located
near field edges (see Section $\S 4.4$). In Jayasinghe et al. (2018) these
were removed by several candidate selection criteria that were relaxed when we
selected variable candidates for this project.

##  3 $g$ \- Band Machine Learning Classifier

The machine learning classifier used for ASAS-SN’s $V$-band variable catalogs
is extensively described in Jayasinghe et al. (2019b). It was based on a
`scikit-learn` (Pedregosa et al., 2018) random forest model that was trained
to distinguish between broad variable types using features which included
light curve statistics, Gaia distances, and multi-band photometry. While this
classifier was extremely accurate at identifying common variable types, it
often mislabeled rare phenomena and light curves with systematic errors.

We retrained the random forest classifier described in Jayasinghe et al.
(2019c) using features from our new $g$-band data. The training set for this
updated classifier is the same as that used previously. We included two
additional features based on the Lafler-Kinmann (Lafler & Kinman, 1965;
Clarke, 2002) string length statistic (LKSL). We calculated the LKSL statistic
$T(t)$ on the temporal light curve using the definition

$T(t)=\frac{\sum_{i=1}^{\rm N}(m_{i+1}-m_{i})^{2}}{\sum_{i=1}^{\rm
N}(m_{i}-\overline{m})^{2}}\times\frac{(N-1)}{2N}$ (1)

from Clarke (2002), where the $m_{i}$ are the time ordered magnitudes and
$\overline{m}$ is the mean magnitude. We also calculated the LKSL statistic
sorting the light curve based on phase for both the best GLS period and twice
the best GLS period, which we will call $T(\phi|P)$ and $T(\phi|2P)$
respectively. For the two new classification features, we used the difference
in the Lafler-Kinmann string length statistics ordered in phase using the best
period and ordered as time,

$\delta(t,P)=\frac{T(\phi|P)-T(t)}{T(t)},$ (2)

and the difference in the statistics phased by the period and twice the
period,

$\delta(P,2P)=\frac{T(\phi|2P)-(T(\phi|P)}{T(\phi|P)}.$ (3)

The ML classification pipeline automatically corrects the period as described
in Jayasinghe et al. (2019d). The updated RF classifier classifies sources
into 7 broad classes (CEPH, DSCT, ECL, LPV, RRAB, RRc/RRd, and ROT) which are
subsequently refined into sub-classes (see Jayasinghe et al. 2019c). The
overall precision, recall and $F_{1}$ parameters for the updated RF classifier
are 94.4$\%$, 95.3$\%$ and 94.7$\%$ respectively. An important feature of the
ML classifier to keep in mind is that it only provides a probability for the
type of variable. There is no equivalent of the "Junk" class available to the
citizen scientists in large part because there was no training set to define
it.

Figure 2: _Citizen ASAS-SN_ workflow for classifying periodic variables.
(Left) 2 phased light curves and the observed light curve. (Right) Possible
classifications for users to select. This variable would best be classified as
an Eclipsing Binary.

##  4 Project Description and Results

Table 1: Breakdown of the number of most voted classifications for each variable type, including those found the AAVSO VSX (Watson et al., 2006), OGLE III (Poleski et al., 2012), and OGLE IV (Kozlowski et al., 2013) catalogs. The variables listed as unknown are those with nonspecific classifications such as MISC or VAR. | Total Candidates | Eclipsing Binaries | Pulsating Variables | Rotational Variables | Unknown Variables | Junk
---|---|---|---|---|---|---
N Candidates | 40640 | 12292 | 11621 | 4529 | 161 | 12037
In VSX | 16750 | 9018 | 6230 | 1320 | 29 | 153
In OGLE III | 1132 | 214 | 819 | 43 | 1 | 55
In OGLE IV | 2560 | 464 | 1989 | 75 | 0 | 32
Average Probability | 0.72 | 0.77 | 0.72 | 0.53 | 0.40 | 0.75
New | 10420 | 2923 | 4234 | 3132 | 131 | 0
Figure 3: The normalized confusion matrix between the citizen science voted
classifications on the horizontal axis and the true class classification based
on the AAVSO VSX, OGLE III, and OGLE IV catalogs.

Volunteers working on _Citizen ASAS-SN_ are shown images with light curves
phased by both the best GLS period and twice the best GLS period along with
the observed light curve. Our goal was to address several simple but common
problems distinguishing between variable types (such as RRc RR Lyrae variables
and EW eclipsing binaries). For eclipsing binaries, the best period returned
by the GLS periodogram is often 1/2 of the orbital period, which is why the
light curve phased with twice the best GLS period is shown. When phased with
the correct orbital period, the light curves of eclipsing binaries show a
distinct separation of the primary and secondary eclipses allowing for their
accurate classification; this behavior is shown in the example light curve
shown in Figure 2. The observed light curve is useful for identifying long-
period variables such as Miras, and evolving variables like rotating spotted
stars.

We designed our project workflow to be easy to navigate and accessible to a
wide array of volunteers. Because we expect no prior knowledge of variable
star classification, we first present users with a tutorial that details the
classification process and summarizes the science. Volunteers also have access
to a field guide that describes common variable stars and their light curves.
Our workflow tasks users to determine the correct basic classification,
selecting between three broad classes (Pulsating Variables, Eclipsing
Binaries, Rotational Variables), choosing the option “Unknown Variable” for
ambiguous cases, or flagging the light curve as “Junk”. Figure 2 shows an
example of the workflow. As users get started, we present them with a variety
of “gold-standard” (GS) candidates that have been classified by the science
team. These GS variables provide the user with feedback on their
classifications to train them in the process. As users make more
classifications, GS variables become less frequent and the user begins to
classify new light curves.

We released Citizen ASAS-SN for public use on January 5th, 2021, and it has
since accrued over 3,000 volunteers and $\sim$800,000 classifications. We
launched the project with a set of 40,640 variable candidates around the South
celestial pole ($\delta<-60^{\circ}$). In addition to the basic
classification, our users pointed out many interesting variables on the
project’s Talk forum. We designed the workflow so that a variable candidate
stops being shown to users once it has reached a retirement limit of 10 votes.
If the number of “Junk” votes reaches 5, then the candidate is retired early.
Once every subject was retired, we tallied up the number of votes each
candidate received in each category. We then assigned each candidate a “most
voted label” which describes the most popular variable type chosen by our
volunteers. If a tie occurred for the most popular vote, the most voted class
was chosen randomly between the tied options.

A breakdown of the most voted class for each retired variable is shown in
Table 1. Of the three main variability classes (Pulsating Variables,
Rotational Variables, and Eclipsing Variables), the Rotational Variable class
was voted the least common, making up only 12% of the total. Pulsating
variables, eclipsing binaries, and junk variable classifications were 28%,
29%, and 30%, with less than $1\%$ classified as Unknown.

###  4.1 Cross-matches to external catalogs

We cross-matched our initial subject set of 40,640 candidates with previously
classified variables stars in the AAVSO VSX (Watson et al., 2006), OGLE III
(Poleski et al., 2012), and OGLE IV (Kozlowski et al., 2013) catalogs using a
matching radius of 16 arcsec and found 16,750 matches in VSX, 1,132 matches in
OGLE III, and 2,560 matches in OGLE IV. The VSX catalog contains all the
variables previously identified by ASAS-SN (Jayasinghe et al., 2020). After
excluding the Junk classifications, our volunteers discovered 10,420 new
variables. Known eclipsing binaries made up the majority of the matches with
VSX, while pulsating variables appear to be the most common match in the OGLE
catalogs. A breakdown of the number of candidates accounted for by VSX, OGLE
III, and OGLE IV candidates is shown in Table 1 along with the full candidate
set. Table 1 also gives the average probability defined as the ratio between
the number of votes for the most popular classification and the total number
of votes which we will refer to as the classification strength.

Our volunteers were able to recover 99%, 95%, and 99% of the previously
cataloged VSX, OGLE III, and OGLE IV variables respectively. If we define the
classifications in these catalogs as the “true class” and the the most popular
_Citizen ASAS-SN_ classification as the “voted class”, we find the confusion
matrix shown in Figure 3. Our users could reliably distinguish between the
three broad variability types, with pulsating variables as the most
identifiable. Overall we found that our users correctly identified 81% of
known eclipsing binaries, 89% of known pulsating variables, and 49% of known
rotational variables. The poor performance on the rotational variables was
expected because the morphology of their light curves can vary widely, leading
to inconsistent classifications (e.g., Thiemann et al. 2021).

Figure 4: Normalized distribution of classification strengths for each
variable class.

###  4.2 User Performance

For our first set of candidates, a total of 2,298 volunteers participated in
_Citizen ASAS-SN_ and they made 403,626 classifications. Of these, 370,277
were of the variable candidates and 33,349 were of the gold-standard
variables. We found that 1,594 users made classifications from accounts
registered with the Zooniverse platform, while 704 users made classifications
from unregistered accounts. The registered users contributed to 95% of the
total classifications, while unregistered users contributed 5% of
classifications.

Figure 5: Confusion matrices for known variable classes against voted classes
binned by classification strength (P).

The next metric we considered was how correlated our user’s votes were with
each other for each variable type. To do this, we computed a “Classification
Strength” $P$ as the ratio between the number of votes for the voted
classification type and the total number of votes. This metric would be
$P=1.0$ if all user classifications agree for a particular variable candidate.
There is a lower bound of $P=0.2$, where the 10 votes were evenly divided over
the five possible classifications. In Figure 4, we show the distribution of
classification strengths for each variable class. The mean classification
strength was highest for eclipsing, pulsating, and junk variables with
averages of $\langle P\rangle$ $=$ 0.78, 0.73 and 0.74 respectively. For
candidates most voted as Rotational Variable and Unknown Variable, there are
more disagreements between users with mean classification strengths of
$\langle P\rangle$ $=$ 0.53 and 0.37 respectively. The low classification
strength for rotating variables is in agreement with the poor performance
shown in Figure 3. Given the nature of unknown variable types, a low
classification strength is to be expected, as the class was designed to
encapsulate difficult to classify variables and anomalous light curves.

In Figure 5, we show the confusion matrix (see Figure 3) for 4 ranges of
classification strength. As the classification strengths increase, the
performance of the citizen scientists improves for the eclipsing, pulsating
and junk categories. But for rotational variables, a higher classification
strength does not translate into better performance. In fact, their
performance was worst in the higher classification strength bin, although this
could be a statistical fluke because few rotational variables had such high
classification strengths.

Figure 6: Light curves for a random sample of pulsating variables, with
classification probabilities of 1.0, meaning all classifiers agreed on the
variable type. The machine learning classification and its probability $P_{\rm
ML}$ are given in the upper right corner. Figure 7: Light curves for a random
sample of rotational variables, with classification probabilities of 1.0,
meaning all classifiers agreed on the variable type. The machine learning
classification and its probability $P_{\rm ML}$ are given in the upper right
corner. Figure 8: Light curves for a random sample of eclipsing binaries, with
classification probabilities of 1.0, meaning all classifiers agreed on the
variable type. The machine learning classification and its probability $P_{\rm
ML}$ are given in the upper right corner. Figure 9: Light curves for
candidates with low probability classifications; classification probability <
0.5. Users found these light curves difficult to classify. The machine
learning classification and its probability $P_{\rm ML}$ are given in the
upper right corner.

Light curves for variable candidates with high classification probabilities (P
= 1.0) are shown in Figures 6, 7, and 8. Our users all agreed on their
classifications of these candidates and found them easy to classify. We show
examples of candidates that our users had difficulty classifying (i.e., with
low classification probabilities, P < 0.5) in Figure 9. Sources with low
classification probabilities typically displayed atypical pulsation patterns
or were near our detection limits.

###  4.3 Grading

Figure 10: Distribution of the users in their number of classifications and
their grade defined by the fraction of time they joined the majority vote.
Histograms show the projected distribution of each quantity. The dashed lines
show the division of the users into the inactive ($N_{\text{Class}}\leq 14$),
active ($N_{\text{Class}}$ > 14), and exceptional ($N_{\text{Class}}>14$ and
grade > 0.5) groups.

The average user of Citizen ASAS-SN made $\sim 17$ classifications, of which
$\sim 3$ were for Gold Standard (GS) targets and $\sim 14$ were for our new
candidates. The distribution of the total number of new candidate
classifications made by each volunteer is shown in Figure 10. We graded each
user based on the number of times they agreed with the most popular
classification as a proxy for the correct classification. Users with very few
classification submissions produce the peaks at 0.0, 0.5, and 1.0. We divided
our users into active and inactive groups, where an active user was one who
submitted more than the median number of non-GS classifications $N>14$. The
distribution of the grades and total classification count for the users is
shown in Figure 10.

Figure 11: Normalized grade distributions for active users and inactive users. Active users are defined as users who made more than the median number of classifications ($N>14$). We excluded inactive users with less than 3 classifications to lessen the peaks at 0.0, 0.5, and 1.0. Table 2: Breakdown of user grades and candidate classification counts for all, active ($N_{\text{Class}}$ > 14), and inactive users ($N_{\text{Class}}\leq 14$). | $N_{\text{Users}}$ | Median Grade | Median $N_{\text{Class}}$ | Total $N_{\text{Class}}$
---|---|---|---|---
All Users | 1982 | 0.60 | 14 | 370277
Active Users | 975 | 0.63 | 59 | 365652
Inactive Users | 1007 | 0.50 | 3 | 4625

While 2,298 users classified objects, only 1,982 classified some of the new
candidates and the remaining only looked at GS targets. We only assigned
grades to users who classified non-GS light curves. Of these, 975 were active
and 1,007 were inactive. Although inactive users were the majority, they
contributed a negligible number of classifications. Of the 370,277 candidate
classifications, 365,652 (99%) were made by active users. The median number of
classifications made by active users was 59 while inactive users had a median
of 3. Table 2 summarizes the user performance for all, active, and inactive
users. As shown in Figure 11, the active members of the project outperformed
the inactive group in terms of voter agreement, with active users receiving a
median grade of 63% compared to 50% for inactive users.

Figure 12: Radial projection of variable candidates around the South
equatorial pole (top, left), variable candidates not voted as Junk (top,
right), cross-matches to known VSX, OGLE III, and OGLE IV variables (bottom,
left), and new variables (bottom, right). The concentric rings seen in the top
left panel are due to spurious variables along field edges. Figure 13: (Left)
Confusion matrix for inactive and low graded users; users with a grade $\leq
0.5$ or $N\leq 14$ classifications. (Right) Confusion matrix for high graded
active users; users with a grade $>0.5$ and $N>14$ classifications.

In Figure 10, the grades of the active users appear to increase with the
number of classifications and there is less scatter. There also appears to be
a lower bound to this distribution proportional to log$(N_{\text{Class}})$. We
defined the 766 active users ($N_{\text{Class}}>14$) with grades greater than
0.5 as exceptional. Figure 13 compares the confusion matrices for the
exceptional users to the inactive ($N_{\text{Class}}\leq 14$) or poor active
($N_{\text{Class}}>14$ and grade $\leq$ 0.5) users. The highly graded active
users were much better at correctly classifying known variable stars compared
to inactive and low scoring users. The better performance is presumably a
combination of learning, interest, and motivation.

###  4.4 New Discoveries

Figure 14: (Left) Confusion matrix for machine learning classifications
against the most voted class. (Right) for machine learning classifications
against the classification of known variable stars.

After fully classifying these new ASAS-SN variable candidates in the southern
sky, our users have helped us discover over 10,000 new variable sources that
are not present in the existing VSX, OGLE III and OGLE IV variable star
catalogs. A breakdown of the number of new variables and their most voted
variable types are shown in Table 1.

When narrowing down our initial 40,640 candidates sources, we first removed
all candidates that were voted as “Junk” by our users. Figure 12 shows the sky
distribution of our full candidate set and the confirmed variables. The full
candidate set displays concentric rings of artifacts associated with the lower
signal to noise field edges created by the vignetting of the telescopes.
Retrospectively, we also found that they mostly had periods of $\sim 1$ day or
$\sim 1$ lunar month and $g$ magnitudes near our detection limits (see Figure
1). In the $V$-band catalog (Jayasinghe et al., 2020), these were being
automatically rejected because sources very close to these periods were not
considered as candidates, but we had dropped this restriction when selecting
candidates for this study. We found that the candidates producing the
concentric ring pattern were systematically classified as Junk, and there are
no patterns in the sky maps once the Junk candidates are removed. This shows
that citizen science is an effective tool for cleaning datasets of false
positives. We have, however, added selection criteria so that these false
positives are now automatically removed (see §5).

After removing the Junk candidates, we cross-matched our sample with existing
variable star catalogs to identify the known variables. This resulted in a
sample of 10,420 new ASAS-SN discoveries. The positions of these new variable
stars on the sky are also shown in Figure 12, along with the candidate, non-
junk, and cross-matched sets. Of the new variable sources, the biggest subset
was pulsating variables with 4,234 found by our users. Rotational variables
were the next most common with 3,132 sources, and eclipsing binaries were the
least common with 2,923 sources. Our users also classified 131 of the new
variables sources as unknown variables with difficult to classify light
curves.

##  5 Machine Learning and Citizen Science

Using the updated RF classifier, we classified the candidate set and and
separated the outputs into Junk and non-Junk groups. We compare the machine
learning classifications of the $\sim$28,000 non-Junk candidates and the
variables with known classifications in Figure 14. The comparison between the
machine learning classifications and the most voted class by our users shows
the same pattern as in Figure 3. The $g$-band RF classifier agreed with our
users’ classifications $77\%$, $90\%$, and $55\%$ of the time for eclipsing,
pulsating, and rotating variables respectively. Figure 14 also shows a
confusion matrix comparing the machine learning classifications to the
classifications for known variables. Here the agreement is much stronger at
$96\%$, $93\%$, and $82\%$ for eclipsing, pulsating, and rotating variables
respectively. Compared to our citizen scientists, the $g$-band classifier was
much more efficient at classifying known variable stars in our candidate
sample. We recognize that this is a bit circular as some of the known
variables were either used to train the ML classifier or classified by the
ASAS-SN $V$-band machine learning classifier. Using the more refined
classifications from the $g$-band classifier, we show the $M_{G}$ v.s.
$G_{BP}-G_{RP}$ color-magnitude diagram and the $M_{G}$ v.s. $G_{BP}-G_{RP}$
period-luminosity diagram for the non-junk candidates broken down by type in
Figure 15. The positions for each subclass of variables agrees with the
distribution of variable stars in the ASAS-SN $V$-band catalog Jayasinghe et
al. (2019c).

Figure 15: The Gaia EDR3 $M_{G}$ v.s. $G_{BP}-G_{RP}$ color-magnitude diagram (Left) and the $M_{G}$ v.s. $\text{log}_{10}(P/days)$ period luminosity diagram (Right) for our set of non-Junk variables using labels given by the $g$-band classifier. Figure 16: A distribution of the classifications given to the Junk and non-Junk sources by the $g$-band classifier. Figure 17: (top,left and middle) Distribution of the best probabilities assigned by the $g$-band classifier to each source in the Junk and non-Junk samples. (top,right) Distribution of the fraction of non-Junk sources viewed as real by the ML classifier as a function of classification probability. (bottom) Distribution of the cumulative fraction of Junk and non-Junk variables as a function of their ML Junk probability. Table 3: ML Classification breakdown of the real _Citizen ASAS-SN_ variables RF Classification | Description | Broad VSX Type | $N_{\text{tot}}$ | $N_{\text{new}}$ |  $N$ (Prob >0.9) |  $N$ (Prob >0.5)
---|---|---|---|---|---|---
CWA | W Virginis type variables with P > 8 d | Pulsating Variable | 206 | 20 | 111 | 182
CWB | W Virginis type variables with P < 8 d | Pulsating Variable | 996 | 15 | 90 | 613
DCEP |  $\delta$ Cephei-type/ classical Cepheid variables | Pulsating Variable | 194 | 35 | 87 | 164
DCEPS | First overtone Chepheid variables | Pulsating Variable | 64 | 7 | 4 | 46
DSCT |  $\delta$ Scuti type variables | Pulsating Variable | 700 | 510 | 348 | 679
EA | Detached Algol-type binaries | Eclipsing Binary | 3729 | 991 | 3151 | 3675
EB |  $\beta$ Lyrae-type binaries | Eclipsing Binary | 3074 | 867 | 1936 | 2873
EW | W Ursae Majoris type binaries | Eclipsing Binary | 8096 | 2315 | 5202 | 7601
GCAS | Rapidly rotating early type stars | Other | 74 | 27 | 0 | 15
HADS | High amplitude $\delta$ Scuti type variables | Pulsating Variable | 469 | 195 | 252 | 455
L | Irregular Variables | Other | 556 | 155 | 475 | 538
M | Mira variables | Pulsating Variable | 142 | 13 | 141 | 141
ROT | Spotted Variables with rotational modulation | Rotational Variable | 4735 | 2964 | 1748 | 4140
RRAB | RR Lyrae variables with asymmetric light curves | Pulsating Variable | 3461 | 1292 | 2719 | 3335
RRC | First Overtone RR Lyrae variables | Pulsating Variable | 1246 | 540 | 897 | 1185
RRD | Double Mode RR Lyrae variables | Pulsating Variable | 103 | 63 | 84 | 100
RVA | RV Tauri variables (Subtype A) | Pulsating Variable | 1 | 1 | 0 | 0
SR | Semi-regular variables | Pulsating Variable | 172 | 58 | 129 | 164
SRD | Semi-regular variables (Subtype D) | Pulsating Variable | 3 | 0 | 0 | 1
YSO | Young Stellar Objects | Other | 31 | 13 | 4 | 19
VAR | Variable star of unspecified type | Other | 551 | 339 | 24 | 229

The ML classifier assigns a probability for each light curve to be a
particular type of variable, and we adopt the highest probability
classification and the frequencies of these classifications for the Junk and
non-Junk sources are shown in Figure 16. The type distributions of the Junk
and non-Junk sources are quite different, with many of the Junk sources placed
in the non-specific VAR class. As shown in Figure 17 the ML classification
probabilities for the Junk and non-Junk stars are also very different - the
classification probabilities of the non-Junk sources are strongly peaked near
unity with a median $P_{best}=0.95$, while the Junk sources had a median of
$P_{best}=0.51$. We investigated the Junk sources with high ML probabilities
and generally agreed with the citizen scientists, although there were some
real but low amplitude variables. The treatment of the Junk sources by the ML
classifier illustrates a standard shortcoming of machine learning. As trained,
it has to classify every light curve as a variable, but for Junk sources it
"compensates" by having low classification probabilities and by putting most
of them into the least well-defined variable type (generic VAR).

Figure 18: (Left) Observed and phased (Middle and Right) light curves for ASASSN-V J085305.34-824360.0. Table 4: Breakdown of the variable stars shown in Figures 18, 19, and 20. ID (ASASSN-V) | RA | Dec | Voted Class | $P_{\rm CS}$ | ML Class | $P_{\rm ML}$ | Period | Other ID
---|---|---|---|---|---|---|---|---
J173255.51-621040.8 | 263.23129058 |  $-$62.17799005 | Eclipsing Binary | 0.6 | EA | 0.725 | 8.215 | ASASSN-V J173255.51-621040.7
J115419.17-613118.0 | 178.57987727 |  $-$61.52166326 | Pulsating Variable | 0.5 | DCEP | 0.364 | 32.81 | ASAS J115419-6131.3
J054112.57-692608.7 | 85.30236232 |  $-$69.43573717 | Eclipsing Binary | 0.5 | EW | 0.431 | 8.582 | OGLE-LMC-ECL-22442
J005436.05-702535.5 | 13.65019648 |  $-$70.42652394 | Eclipsing Binary | 0.8 | EW | 0.553 | 0.365 | WISE J005436.0-702535
J193359.68-680634.9 | 293.49868276 |  $-$68.10970064 | Eclipsing Binary | 0.4 | ROT | 0.811 | 5.137 | ASAS J193359-6806.6
J085305.34-824360.0 | 133.2722443 |  $-$82.73333148 | Eclipsing Binary | 0.7 | EB | 0.658 | 20.39 | ASAS J085305-8244.0
J090020.74-644127.9 | 135.08641928 |  $-$64.69108615 | Pulsating Variable | 0.8 | EW | 0.566 | 0.289 | N/A (New Discovery)

One approach to a solution would be to simply use the mismatched distribution
in classification probability to try to automate the elimination of the Junk
sources. Figure 17 (upper right panel) also shows the fraction of non-Junk
sources as a function of the classification probability. The variable sample
can be made very pure, but such a sample would also be quite incomplete. For
example, if we simply keep things with classification probabilities greater
than the median probability for the Junk sources, we lose 10% of the real
variables while still have half of the junk sources.

A better ML solution is to use the availability of Junk and non-Junk training
sets to train a new random forest classifier to distinguish them. We split the
DR1 sample and used 40% of it for training and 60% for testing. The resulting
classifier had an F1 score of 95.4% and precision/recall scores for non-Junk
and Junk sources of 98%/92% and 96%/96%, respectively. The bottom panel of
Figure 17 shows the distribution of the sources in the Junk classification
probability. Keeping only sources with a less than 50% Junk classification
probability eliminates roughly 97% of the Junk while losing only 3% of the
variables. The upper right panel of Figure 17 shows the sample purity as
function of the original variable classification probability for various cuts
on the Junk probability. Clearly the path forward is to fully incorporate a
Junk class into the g-band ML classifier.

##  6 Unusual Variables

Figure 19: A sample of interesting light curves our users pointed out on the
project’s Talk forum. Figure 20: (Left) Observed and phased (Middle and Right)
light curves for ASASSN-V J090020.74-644127.9.

When users encounter strange light curves or sources they found difficult to
classify, they can post a comment about the source to the project’s Talk
forum. Once posted, the particularly interesting variables led to considerable
discussion. The Zooniverse platform allows any user to search for specific
tags, which makes the identification of weird variables relatively easy. There
were 330 instances of light curves described as “interesting”, 364 described
as “unusual” and 92 described as “weird”. We show several examples of such
variables in Figures 18, 19, and 20. Each of these variables was extensively
discussed in the Talk forum because of their bizarre light curves. Table 4
shows the ML classification breakdown for each variable shown in this section.

Many users flagged light curves that have recurrent outliers which indicate
the presence of competing sources of variability. These systems are of
interest because stars that exhibit multiple pulsation behaviors can act as
stellar laboratories, so their identification for additional follow-up is
important (Thiemann et al., 2021). An example of such a system is ASASSN-V
J085305.34-824360.0 (see Figure 18). On the project, we displayed this
candidate using the period $P=10.19$ days. This caught the attention of our
users because the light curve displayed a strong sinusoidal variation with
recurring outliers at each minima. The regular nature of these outliers
indicated that this system might be an eclipsing binary. Additionally, the
observed light curve shows distinct amplitude modulation, likely due to
spotting on the surface. This star was classified as an eclipsing binary (EB
type) in the $V$-band (ASASSN-V J085305.74-824401.0) with a classification
probability of 0.962 and period of 20.4 days. We found the correct period to
be 1/2 of this at $P=10.21$ days. When phasing the observed light curve with
this period, the primary and secondary eclipses become visible, while the
rotational signature is blurred. This period is very close to the best GLS
period presented to our users on _Citizen ASAS-SN_ , which suggests that this
system is a nearly synchronized eclipsing binary with active spotting.

Other particularly interesting and rare systems are pulsating variables in
eclipsing binaries. These systems are powerful tools because they allow
researchers to derive the fundamental stellar parameters and probe the
internal structure of stars (Kahraman Aliçavuş et al., 2017). The system
ASASSN-V J090020.74-644127.9 is an example where there is a pulsation period
of 0.14 days (probably a HADS variable), and a 5.04 day period for the
eclipses. We only presented users with the phased light curves for the
pulsation behavior but users pointed out that there may be a hidden eclipse.
We believe that the identification of strange variables for additional
followup is one of the key strengths of citizen science. A more in-depth
analysis of the odd light curves and the hybrid systems will be done in future
papers.

##  7 Conclusions

We present the first results of _Citizen ASAS-SN_. This includes the analysis
of 403,626 classifications of 40,640 variable candidates at the south
celestial pole ($\delta<-60^{\circ}$) from 2,298 users. Classifications for
these variables were made between January 5th and March 27th 2021. The final
results are available at the ASAS-SN Variable Star Database (https://asas-
sn.osu.edu/variables). The primary classification for each variable comes from
the machine learning classifier, but we include the most popular citizen
science classification and its probability. Future updates will be released as
we move up the sky in declination. Table 3 lists the number of sources of each
variable type in the catalog along with the number of new discoveries in each
category.

If we compare the user classifications to either published (VSX, OGLE) or our
ML classifications, we found that our volunteers classified eclipsing binaries
and pulsating variables most consistently, while struggling to classify
rotational variables. We also found that it was exceedingly rare for known
variables to be misclassified as Junk, accounting for less than $\sim 2\%$ for
each variable type. We calculated a probability metric for each variable
candidate that measures the agreement between users. We found that our users
were likely to agree on the classifications for candidates that were most
voted as eclipsing binaries, pulsators, and Junk variables. Classifications
for variable candidates most voted as rotational and unknown variables were
more difficult for our users to agree upon.

User activity generally correlated with higher classification accuracy and
higher user agreement, showing that experience improves performance. Our
citizen scientists discovered 10,420 new variable sources including, as they
defined them, 4,234 pulsating variables, 3,132 rotational variables and 2,923
eclipsing binaries with 131 candidates flagged as Unknown. In addition to
these new sources, many users have pointed out unusual or extreme variable
candidates on the _Citizen ASAS-SN_ Talk forum for additional follow-up.
Moving forward, we plan to release subsequent candidate moving North in
declination. We also plan to extend our workflow to cover higher order
classifications including irregular variable stars.

We also built a new $g$-band machine learning classifier trained on light
curves features from variables in our $g$-band catalog. We found that our
classifier was more accurate at classifying known variables than our users.
However, the citizen scientists out performed the classifier when it came to
identifying Junk light curves. The ML classifier does assign them lower
classification probabilities and classifies many of the to the generic VAR
class. As built, the ML classifier has to assign all light curves to some type
of variable because it has no Junk output class. We now have a Junk training
set, and a simple ML classifier simply built to distinguish Junk and non-Junk
sources performed very well. We now use this initial Junk classifier to purge
these candidates before releasing new light curves to Citizen ASAS-SN. Moving
forward, we will rebuild the overall ML variable classifier to include a Junk
classification. While it was not one of our initial goals, the construction
and continued expansion of a Junk training set will be a very valuable
contribution of Citizen ASAS-SN.

## Acknowledgements

We thank the Zooniverse team and each volunteer that participated in _Citizen
ASAS-SN_. We thank the Las Cumbres Observatory and their staff for its
continuing support of the ASAS-SN project. We also thank the Ohio State
University College of Arts and Sciences Technology Services for helping us set
up and maintain the ASAS-SN variable stars database.

The ASAS-SN team at OSU is supported by the Gordon and Betty Moore Foundation
through grant GBMF5490 to the Ohio State University, and NSF grant
AST-1908570. CSK and KZS are also supported by NSF grant AST-1814440.
Development of ASAS-SN has been supported by NSF grant AST-0908816, the Mt.
Cuba Astronomical Foundation, the Center for Cosmology and AstroParticle
Physics at the Ohio State University, the Chinese Academy of Sciences South
America Center for Astronomy (CAS- SACA), the Villum Foundation, and George
Skestos. TAT is supported in part by Scialog Scholar grant 24216 from the
Research Corporation. Support for JLP is provided in part by FONDECYT through
the grant 1151445 and by the Ministry of Economy, Development, and Tourism’s
Millennium Science Initiative through grant IC120009, awarded to The
Millennium Institute of Astrophysics, MAS. SD acknowledges Project 11573003
supported by NSFC. Support for MP and OP has been provided by the
PRIMUS/SCI/17 award from Charles University.

This work has made use of data from the European Space Agency (ESA) mission
Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing
and Analysis Consortium (DPAC,
https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has
been provided by national institutions, in particular the institutions
participating in the Gaia Multilateral Agreement.

This research has made use of the VizieR catalogue access tool, CDS,
Strasbourg, France. The original description of the VizieR service was
published in A$\&$AS 143, 23.

This research made use of Astropy, a community-developed core Python package
for Astronomy (Astropy Collaboration, 2013).

## Data Availability

The variables are publicly cataloged with the AAVSO and the ASAS-SN light
curves can be obtained using the ASAS-SN Sky Patrol (https://asas-sn.osu.edu).
The catalog of variables and the associated light curves are available on the
ASAS-SN variable stars database (https://asas-sn.osu.edu/variables). The
external photometric data underlying this article were accessed from sources
in the public domain: Gaia (https://www.cosmos.esa.int/gaia), 2MASS
(https://old.ipac.caltech.edu/2mass/overview/access.html), AllWISE
(http://wise2.ipac.caltech.edu/docs/release/allwise/) and GALEX
(https://archive.stsci.edu/missions-and-data/galex-1/).

## References

  * Alard (2000) Alard C., 2000, A&AS, 144, 363
  * Alard & Lupton (1998) Alard C., Lupton R. H., 1998, ApJ, 503, 325
  * Alcock et al. (2000) Alcock C., et al., 2000, The Astrophysical Journal, 542, 281–307
  * Alhammady & Ramamohanarao (2004) Alhammady H., Ramamohanarao K., 2004, in Fourth IEEE International Conference on Data Mining (ICDM’04). pp 315–318
  * Bellm (2014) Bellm E. C., 2014, The Zwicky Transient Facility (arXiv:1410.8185)
  * Brown et al. (2013) Brown T. M., et al., 2013, Publications of the Astronomical Society of the Pacific, 125, 1031–1055
  * Brown et al. (2018) Brown A. G. A., et al., 2018, Astronomy & Astrophysics, 616, A1
  * Clarke (2002) Clarke D., 2002, A&A, 386, 763
  * Derue et al. (2002) Derue F., et al., 2002, Astronomy & Astrophysics, 389, 149–161
  * Drake et al. (2009) Drake A. J., et al., 2009, The Astrophysical Journal, 696, 870–884
  * Freedman et al. (2019) Freedman W. L., et al., 2019, The Astrophysical Journal, 882, 34
  * Hasanzadeh et al. (2021) Hasanzadeh A., Safari H., Ghasemi H., 2021, Monthly Notices of the Royal Astronomical Society, 505, 1476
  * Heinze et al. (2018) Heinze A. N., et al., 2018, The Astronomical Journal, 156, 241
  * Holoien et al. (2016) Holoien T.-S., et al., 2016, Monthly Notices of the Royal Astronomical Society, 464, 2672–2686
  * Jayasinghe et al. (2018) Jayasinghe T., et al., 2018, MNRAS, 477, 3145
  * Jayasinghe et al. (2019a) Jayasinghe T., et al., 2019a, MNRAS, 485, 961
  * Jayasinghe et al. (2019c) Jayasinghe T., et al., 2019c, MNRAS, 486, 1907
  * Jayasinghe et al. (2019b) Jayasinghe T., et al., 2019b, Monthly Notices of the Royal Astronomical Society, 486, 1907
  * Jayasinghe et al. (2019d) Jayasinghe T., et al., 2019d, Monthly Notices of the Royal Astronomical Society, 491, 13
  * Jayasinghe et al. (2020) Jayasinghe T., et al., 2020, arXiv e-prints, p. arXiv:2006.10057
  * Jayasinghe et al. (2021) Jayasinghe T., et al., 2021, MNRAS, 503, 200
  * Kahraman Aliçavuş et al. (2017) Kahraman Aliçavuş F., Soydugan E., Smalley B., Kubát J., 2017, MNRAS, 470, 915
  * Kochanek et al. (2017) Kochanek C. S., et al., 2017, PASP, 129, 104502
  * Kozlowski et al. (2013) Kozlowski S., et al., 2013, Supernovae and Other Transients in the OGLE-IV Magellanic Bridge Data (arXiv:1301.3909)
  * Lafler & Kinman (1965) Lafler J., Kinman T. D., 1965, ApJS, 11, 216
  * Leavitt (1908) Leavitt H. S., 1908, Annals of Harvard College Observatory, vol. 60, pp.87-108.3
  * Pedregosa et al. (2018) Pedregosa F., et al., 2018, Scikit-learn: Machine Learning in Python (arXiv:1201.0490)
  * Pojmanski (2002) Pojmanski G., 2002, The All Sky Automated Survey. Variable Stars in the 0h - 6h Quarter of the Southern Hemisphere, https://arxiv.org/abs/astro-ph/0210283
  * Poleski et al. (2012) Poleski R., Soszyński I., Udalski A., Szymański M. K., Kubiak M., Pietrzynski G., Wyrzykowski L., Ulaczyk K., 2012, The Optical Gravitational Lensing Experiment. The Catalog of Stellar Proper Motions toward the Magellanic Clouds (arXiv:1203.2649)
  * Prusti et al. (2016) Prusti T., et al., 2016, Astronomy & Astrophysics, 595, A1
  * Riess et al. (2018) Riess A. G., et al., 2018, The Astrophysical Journal, 861, 126
  * Scargle (1982) Scargle J. D., 1982, ApJ, 263, 835
  * Shappee et al. (2014) Shappee B. J., et al., 2014, ApJ, 788, 48
  * Thiemann et al. (2021) Thiemann H. B., Norton A. J., Dickinson H. J., McMaster A., Kolb U. C., 2021, Monthly Notices of the Royal Astronomical Society
  * Tonry et al. (2018a) Tonry J. L., et al., 2018a, Publications of the Astronomical Society of the Pacific, 130, 064505
  * Tonry et al. (2018b) Tonry J. L., et al., 2018b, ApJ, 867, 105
  * Torres et al. (2009) Torres G., Andersen J., Giménez A., 2009, The Astronomy and Astrophysics Review, 18, 67–126
  * Trouille et al. (2019) Trouille L., Lintott C. J., Fortson L. F., 2019, Proceedings of the National Academy of Sciences, 116, 1902
  * Udalski (2004) Udalski A., 2004, The Optical Gravitational Lensing Experiment. Real Time Data Analysis Systems in the OGLE-III Survey, https://arxiv.org/abs/astro-ph/0401123
  * Watson et al. (2006) Watson C. L., Henden A. A., Price A., 2006, Society for Astronomical Sciences Annual Symposium, 25, 47
  * Woźniak et al. (2004) Woźniak P. R., et al., 2004, The Astronomical Journal, 127, 2436–2449
  * Zechmeister & Kürster (2009) Zechmeister M., Kürster M., 2009, A&A, 496, 577


# Multilinguals at SemEval-2022 Task 11: Complex NER in Semantically Ambiguous
Settings for Low Resource Languages

Amit Pandey*, Swayatta Daw*, Narendra Babu Unnam  Vikram Pudi
International Institute of Information Technology, Hyderabad, India
{amit.pandey<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

We leverage pre-trained language models to solve the task of complex NER for
two low-resource languages: Chinese and Spanish. We use the technique of Whole
Word Masking (WWM) to boost the performance of masked language modeling
objective on large and unsupervised corpora. We experiment with multiple
neural network architectures, incorporating CRF, BiLSTMs, and Linear
Classifiers on top of a fine-tuned BERT layer. All our models outperform the
baseline by a significant margin and our best performing model obtains a
competitive position on the evaluation leaderboard for the blind test set.
11footnotetext: Equal Contribution

##  1 Introduction

The Named Entity Recognition (NER) task aims to identify the named entities in
an input sequence and categorize them into certain predefined categories or
class labels. The task of NER can be further broken down into two subtasks: 1)
identification of the entity span and 2) classification of the identified
entity span into predefined class labels. For example, in the sentence: New
York City is the most densely populated major city in the United States., New
York City is a named entity of type LOCATION with an entity span of 3 tokens.

The most popular NER task in the English language is CoNLL Baevski et al.
(2019), which is widely used as a benchmark for most NER models. Multiple
models have been able to obtain sufficiently high performances in this task
setting Wang et al. (2021a); Zhou and Chen (2021); Luoma and Pyysalo (2020);
Schweter and Akbik (2020); Ye et al. (2021); Yamada et al. (2020); Wang et al.
(2021b). The CoNLL training set consists of 14,987 train sentences, which
comprise 203,621 tokens in total for English data. The entity space consists
of 4 different types of entity type labels (locations, persons, organizations,
and miscellaneous) to classify each named token. The English data was taken
from the Reuters Corpus, which comprises of Reuters News Stories for one year.
The training data source, and by extension the labeled named entities,
comprises of majorly popular entities found in the general English textual
content prevalent in the media. Hence, these entities were easier to classify
into the correct classes due to the large prevalence of training data. With
the use of pre-trained transformer-based language models, which are already
trained on a large unlabelled corpus of English text, this task became even
less challenging, as the nature of textual structure in these corpora largely
overlap with that of CoNLL.

However, this task becomes challenging in practical settings where a multitude
of varieties of named entities is possible. Many of these entities, like
Creative Works (CW) and Products (PROD) have complex and ambiguous textual
structural content. Such complex named entities rarely appear even in the
large training data sets, and the length and structure of the named entities
keep changing.

We investigate the task of complex, semantically ambiguous, and low-resource
NER Malmasi et al. (2022b). This task is based on the complex NER, search
query and code-mixing NER challenges introduced by Meng et al. (2021) and
Fetahu et al. (2021). The shared task of MultiCoNER (stands for multilingual
complex NER) adds additional challenge by introducing rarer label types (like
Creative Work, Product, etc.).

Another way to increase the difficulty of NER task is to perform it for low-
resource languages. There is a significant dearth of both labeled and
unlabelled data for such languages. The complexity is further enhanced by
using rarer entity types in such languages. Therefore, the scarcity of
training data, along with the rarity of entity types, makes it difficult for
the models to perform better in the low-resource setting. The shared task of
MultiCoNER introduces datasets in multiple low-resource languages.

We leverage large pre-trained language models trained on low-resource language
corpora to obtain competitive performances in the low-resource and complex NER
setting. We show that simpler architectures successfully outperform other
heavier counterparts. We use standard BERT+CRF-based models to obtain high
performances in the evaluation set. We experiment on two low-resource
datasets: Spanish and Chinese. The code is available at
https://github.com/AmitPandey-Research/Complex_NER

We compare multiple architectures on the test and validation set of the shared
task. The organisers provide a baseline of XLM-RoBERTa model Conneau et al.
(2019) finetuned on the training dataset of the specified language for the
given task. In this paper, we treat the finetuned XLM-RoBERTa as baseline (two
separate models are trained for Spanish and Chinese language) and compare the
performances against our models. All our models beat the baseline by a
significant margin. We describe the prior research work done with respect to
both general and low-resource NER tasks in Section 2. We provide the formal
task description in Section 3, the dataset details in Section 4, the method
and the model architecture in Section 5. We provide details about the
experimental implementation in Section 6. We discuss the results obtained and
error analysis in Sections 7 and 8 respectively, and finally, we conclude the
paper in Section 9.

##  2 Related Work

The task of low-resource NER has been investigated before by multiple
researchers. This line of research focuses mainly on leveraging the cross-
lingual contextual information obtained from low-resource languages. Feng et
al. (2018) use cross-lingual knowledge transfer to train the NER model for the
low-resource target language. Xie et al. (2018) use bilingual dictionaries to
tackle the task of low-resource NER. Rahimi et al. (2019b) proposes a Bayesian
graphical model approach to improve performance on NER tasks.

NER models often use gazetteers (list of named entities) to improve
performance in NER tasks. Rijhwani et al. (2020) creates soft-gazetteers for
low-resource languages, leveraging English Knowledge Bases. Bari et al. (2019)
focuses on an unsupervised approach for NER for to circumvent the label
scarcity problem in low-resource languages. Rahimi et al. (2019a) leverages
multilingual transfer learning from multiple languages for low-resource NER
tasks. Hedderich et al. (2021) uses distant supervision in the low-resource
setting for NER.

There are multiple approaches that have been undertaken in the recent past to
improve the state-of-the-art in NER tasks. Wang et al. (2020) uses
concatenation of embeddings to outperform the state-of-the-art in NER tasks,
as they infer that concatenation of embeddings leads to a better word
representation. Their method automates the process of finding meaningful
embeddings to concatenate for improved performance. Zhou and Chen (2021)
propose a co-regularization framework for entity extraction comprising of
multiple models with different architectures but different parameter
initializations. This helps to tackle overfitting of large neural network-
based models on low-resource training data for NER. Schweter and Akbik (2020)
use document-level features to improve information extraction on entity-
centric tasks.

NER and Relation Extraction are the core information extraction tasks in NLP.
Ye et al. (2021) models this as a span-pair classification problem, and they
further improve the pair representations by considering the dependencies
between the spans (pairs) by strategically packing the markers in the encoder.
Yamada et al. (2020) proposes a novel entity-aware self-attention framework
for transformer based models for NER. Wang et al. (2021b) extracts document-
level context for sentences for which document information is absent. They
treat the sentence as a query and use a search engine to extract the document
level contextual information. Luoma and Pyysalo (2020) uses multiple
neighbouring sentences as the contextual information for NER.

Pre-trained Language Models for NER : Ever since the introduction of BERT
Devlin et al. (2019), transformer based pre-trained language models have
effectively utilized transfer learning for downstream NLP tasks. NER has been
traditionally modeled as a sequence labeling problem. Huang et al. (2015)
proposed a Bidirectional LSTM with a CRF layer on top for classifying tokens
as entities. Jadhav (2020) use a pretrained BERT model with a CRF layer on top
for performing NER on the DailyHunt news dataset. We use a BERT-based model
with a CRF layer on top and achieve competitive performance on low-resource
NER tasks on multiple languages, beating the baseline by a significant margin
in each case.

##  3 Task Description

In this task, we attempt complex NER for two low-resource languages: Spanish
and Chinese. This task presents additional challenges in the form of test
instances consisting of short search queries and low-context sentences. For
this task, the systems had to identify the B-I-O format Ramshaw and Marcus
(1999) (short for beginning, inside, outside) tags for six NER entity type
labels: 1) Person, 2) Product, 3) Location, 4) Group, 5) Corporation, and 6)
Creative Work. The description of these labels is shown in Table 1.

Label | Description
---|---
PER | Person
LOC | Location
GRP | Group
CORP | Corporation
PROD | Product
CW | Creative Work
Table 1: Entity types in the label space

##  4 Dataset

The MultiCoNER dataset Malmasi et al. (2022a) consists of multiple low-
resource languages. We consider Chinese and Spanish languages in this paper.
For the monolingual track, the participants have to train a model that works
for a single language. We fine-tune the language model on the train set to
obtain predictions on dev and test set. The labels from the blind test set are
not disclosed. The dataset follows a BIO tagging scheme and there are 6 entity
types in the label space. The statistics for the Chinese and Spanish datasets
in the monolingual track for the train and dev set are provided in Table 2.
The total number of test instances for both Spanish and Chinese languages
exceed 150,000.

| Train | Dev
---|---|---
# sentences | 15300 | 800
Table 2: Total sentences in Chinese and Spanish monolingual track

##  5 System Overview

At first, we pre-train the BERT language model on unlabelled corpora for the
target low resource language. For Chinese, we use the strategy outlined by Cui
et al. (2020). BERT uses the WordPiece tokenizer Wu et al. (2016) to split
tokens into smaller fragments. It is easier for the masked language model to
predict these masked fragments. However, for the Chinese textual texture, the
Chinese characters are not formed by alphabet-like symbols, so the WordPiece
tokenizer is unable to split the words into small fragments. Hence, we use the
Chinese Word Segmentation (CWS) tool to split the text into separate words and
then use Whole Word Masking (WWM) strategy for the masked language model
objective. In comparison to masking small fragments, this Whole Word Masking
strategy makes it harder for the model to predict whole masked words, leading
to more robustness.

For the Spanish variant, we adopt the strategy outlined by Cañete et al.
(2020). Similar to Cui et al. (2020), they use the strategy of whole word
masking for pre-training BERT language model on unlabelled Spanish corpus.

We adopt the strategy of finetuning these pre-trained BERT models on the
downstream NER task for each language.

###  5.1 BERT+CRF

Conditional random fields (CRFs) are statistical modeling methods used for
pattern recognition. They are better suited for tasks such as Part-of-Speech
(POS) tagging and NER compared to classifiers based on softmax normalization.
Classifiers based on softmax normalization assume the likelihood of the labels
to be conditionally independent, and this causes label bias. CRF alleviates
this issue of label bias by capturing inter-token dependencies in a graphical
model and learning transition scores in addition to the hidden states. In our
model, we use linear-chain CRF. In linear CRFs, prediction for each token is
dependent only on its immediate neighbors. As shown in equation 2, CRF tries
to maximize the ratio of the probability of an optimal sequence of labels to
the probability sum of all the possible sequences of labels. Since CRF focuses
on the sequence of labels, it can avoid errors like B-PER followed by an
I-PROD. Therefore, based on emission scores provided by BERT layer, we
calculate the log-likelihood of a sequence of labels. Now we explain the steps
involved BERT+CRF architecture that is shown in Figure 1.

Figure 1: BERT+CRF architecture

Firstly, we obtain token-level dense representations using a fine-tuned BERT-
based embedding layer. For an input sequence of tokens
$w=(w_{1},w_{2},w_{3},...,w_{n})$, we obtain the $i$th token representation
$x_{i}$ of dimension $d$, where $d$ is the dimension of BERT embeddings. The
token embedding $x_{i}$ is passed to a dense linear layer to transform the
representation from $d$ to $k$ dimensional space (emission score), where $k$
is the number of labels. We calculate emission scores for all the tokens of
the given sequence. We then pass these emission scores to the CRF layer to
obtain the probability for a sequence of labels. The emission scores, obtained
from the previous layer as $P\in R^{n\times{k}}$, are passed to the CRF layer
whose parameters are $A\in R^{k+2\times{k+2}}$. Element $A_{ij}$ denotes the
transition score from the $i$th to the $j$th label. 2 additional states are
added to the start and end of the sequence. For a series of tokens
$w=(w_{1},w_{2},w_{3},...,w_{n})$ we obtain a series of predictions
$y=(y_{1},y_{2},y_{3},...,y_{n})$.

As described in Lample et al. (2016), the score of the entire sequence is
defined as :

$s(w,y)=\sum_{i=0}^{n}A_{y_{i},y_{i+1}}+\sum_{i=1}^{n}P_{i,y_{i}}$ (1)

The model is trained to maximise the log probability of the correct label
sequence:

$\log(p(y|w))=s(w,y)-\log(\sum_{\tilde{y}\in\textbf{$Y_{w}$}}e^{s(w,\tilde{y})})$
(2)

where $Y_{w}$ are all possible label sequences.

###  5.2 BERT+BiLSTM+CRF

We obtain token-level contextual dense representations (BERT embeddings) using
fine-tuned BERT layer. These embeddings are then passed to a BiLSTM layer
which further extracts bidirectional information from the given sequence of
vectors. The information is encoded in the hidden-state representations of the
BiLSTM. We pass these hidden states to the CRF layer to obtain the likelihood
of a sequence of labels. We use the pre-trained language model to map the
tokens in each input sentence to a dense embedding representation. The BERT-
based dense embeddings are passed to the BiLSTM-CRF layer, which is used to
obtain the predicted label for each token in the entire sequence. More
formally, for a sequence of tokens $w=(w_{1},w_{2},w_{3},...,w_{n})$, we
obtain the $i$th token representation $x_{i}$ of dimension $d$, which is the
dimension of the dense vector representations of the BERT-based embeddings
obtained from the pre-trained language model. The sequence of token embeddings
is taken as an input to the BiLSTM in each time step, and the forward hidden
states
$\overrightarrow{h_{f}}=(\overrightarrow{h_{1}},\overrightarrow{h_{2}},\overrightarrow{h_{3}},...,\overrightarrow{h_{n}})$
and the backward hidden states
$\overleftarrow{h_{b}}=(\overleftarrow{h_{1}},\overleftarrow{h_{2}},\overleftarrow{h_{3}},...,\overleftarrow{h_{n}})$
are concatenated to form the combined hidden state representation
$h=[\overrightarrow{h_{f}},\overleftarrow{h_{b}}]$. The combined hidden state
representation $h\in R^{n\times{m}}$, where $m$ is the total dimension of
BiLSTM, is transformed to a $k$ dimensional space using a linear layer, where
$k$ is the total number of labels. Finally, the CRF layer outputs predicted
sequence of labels.

###  5.3 BERT+Linear

This is the simplest architecture based on fine-tuned BERT layer. The input
token sequence is mapped to a vector space of $d$ dimension using a pre-
trained BERT layer. These embeddings are then passed to a classifier that
consists of two Fully Connected (FC) layers followed by a softmax
normalization function. The classifier maps the $d$ dimensional BERT
embeddings to $k$ dimensions, where $k$ is the number of labels. These $k$
dimensional vectors generated by the fully connected layers are softmaxed to
provide a probability distribution across all labels.

| BERT+CRF | BERT+BiLSTM+CRF | BERT+Linear
---|---|---|---
Class Label | Prec | Rec | F1 | Prec | Rec | F1 | Prec | Rec | F1
LOC | 0.8368 | 0.8796 | 0.8577 | 0.8219 | 0.8759 | 0.8481 | 0.8194 | 0.8613 | 0.8399
PER | 0.9065 | 0.9028 | 0.9047 | 0.9177 | 0.9028 | 0.9102 | 0.8933 | 0.9150 | 0.9040
PROD | 0.6970 | 0.7468 | 0.7210 | 0.7278 | 0.7468 | 0.7372 | 0.6864 | 0.7532 | 0.7183
GRP | 0.7952 | 0.7857 | 0.7904 | 0.7751 | 0.7798 | 0.7774 | 0.8061 | 0.7917 | 0.7988
CW | 0.7965 | 0.7135 | 0.7527 | 0.7654 | 0.7135 | 0.7385 | 0.8107 | 0.7135 | 0.7590
CORP | 0.8657 | 0.8227 | 0.8436 | 0.8397 | 0.7801 | 0.8088 | 0.8529 | 0.8227 | 0.8375
Average | 0.8163 | 0.8085 | 0.8117 | 0.8079 | 0.7998 | 0.8034 | 0.8115 | 0.8096 | 0.8096
Table 3: Results of our models on validation dataset for Spanish language | BERT+CRF | BERT+BiLSTM+CRF | BERT+Linear
---|---|---|---
Class Label | Prec | Rec | F1 | Prec | Rec | F1 | Prec | Rec | F1
LOC | 0.9465 | 0.9365 | 0.9415 | 0.9239 | 0.9312 | 0.9275 | 0.9186 | 0.9259 | 0.9223
PER | 0.8497 | 0.9225 | 0.9084 | 0.8971 | 0.9457 | 0.9208 | 0.8955 | 0.9302 | 0.9125
PROD | 0.8867 | 0.8285 | 0.8566 | 0.8662 | 0.8504 | 0.8582 | 0.8593 | 0.8248 | 0.8417
GRP | 0.7500 | 0.6923 | 0.7200 | 0.7727 | 0.6538 | 0.7083 | 0.6923 | 0.6923 | 0.6923
CW | 0.8265 | 0.8617 | 0.8437 | 0.8556 | 0.8191 | 0.8370 | 0.8370 | 0.8191 | 0.8280
CORP | 0.8615 | 0.8750 | 0.8682 | 0.8808 | 0.8854 | 0.8831 | 0.8883 | 0.8698 | 0.8789
Average | 0.8610 | 0.8527 | 0.8564 | 0.8660 | 0.8476 | 0.8558 | 0.8485 | 0.8437 | 0.846
Table 4: Results of our models on validation dataset for Chinese language

|  Prec |  Rec |  F1
---|---|---|---
Baseline System |  0.764 |  0.763 |  0.767
BERT + Linear |  0.811 |  0.809 |  0.809
BERT+BiLSTM+CRF |  0.807 |  0.799 |  0.803
BERT + CRF |  0.816 |  0.808 |  0.811
Table 5: Comparison of models’ performances with baseline on validation
dataset for Spanish language

|  Prec |  Rec |  F1
---|---|---|---
Baseline System |  0.758 |  0.762 |  0.755
BERT + Linear |  0.848 |  0.843 |  0.846
BERT+BiLSTM+CRF |  0.866 |  0.847 |  0.855
BERT+CRF |  0.861 |  0.852 |  0.856
Table 6: Comparison of models’ performances with baseline on validation
dataset for Chinese language

| BERT+CRF
---|---
Class Label | Prec | Rec | F1
LOC | 0.5768 | 0.6571 | 0.6144
PER | 0.7641 | 0.7739 | 0.7690
PROD | 0.6292 | 0.5141 | 0.5659
GRP | 0.5727 | 0.5560 | 0.5642
CW | 0.5331 | 0.5257 | 0.5294
CORP | 0.6605 | 0.6005 | 0.6291
Average | 0.6227 | 0.6046 | 0.6120
Table 7: Performance of Spanish model on test dataset

| BERT+CRF
---|---
Class Label | Prec | Rec | F1
LOC | 0.6930 | 0.7955 | 0.7407
PER | 0.7952 | 0.6377 | 0.7078
PROD | 0.6853 | 0.7232 | 0.7038
GRP | 0.7254 | 0.4608 | 0.5636
CW | 0.5520 | 0.6798 | 0.6093
CORP | 0.6526 | 0.7361 | 0.6918
Average | 0.6839 | 0.6722 | 0.6695
Table 8: Performance of Chinese model on test dataset

##  6 Implementation Details

We implement all our transformer based models using Pytorch and Huggingface
library. The Chinese language model with the Whole Word Masking (WWM)
objective is trained on the Chinese Wikipedia unlabelled text corpus. We use
the same training corpus of 3 billion unannotated Spanish tokens as Cañete et
al. (2020) to pre-train the BERT language model on Spanish data. We implement
3 models: BERT+CRF, BERT+BiLSTM+CRF, and BERT+Linear, for our low resource NER
task setting. We run our experiments between 1-100 epochs. We find that the
best results are obtained at 10 epochs of training for each model after which
the model starts to overfit. We use a dropout from 0.2 to 0.5 for all models.
We employ Adam optimizer with default parameters for all experiments. We also
experiment with a cyclic learning rate between ${1e^{-4}}$ to ${1e^{-6}}$ to
avoid getting stuck in local minima. The size of each of the FC layers in the
BERT+Linear model is 512. We validate the results of all models using our
validation dataset. The hidden layer size of BiLSTM used in the
BERT+BiLSTM+CRF model is 256.

##  7 Results

We compare the performances of all models in the low-resource setting for both
Chinese and Spanish languages. From tables 3 and 4 we observe that the
BERT+CRF model performs the best across both languages on validation set. We
choose the best performing model to evaluate our results on the blind test
set. The baseline model chosen by the organisers of this task is XLM-RoBERTa
Conneau et al. (2019)(base model). It is pre-trained on 2.5TB of filtered
CommonCrawl data containing 100 languages Conneau et al. (2019). Our approach
beats the baseline by a significant margin and outperforms multiple models in
the competition. We present the precision, recall and F1 scores for all 3
models in the Tables 3 and 4 for Spanish and Chinese language respectively. We
also compare the results between the baseline and our models for the
validation dataset in the Tables 6 and 6.

For the Spanish language, we observe that the BERT+CRF (0.8117 F1 score) beats
BERT+Linear (0.8096 F1 score) by a slender margin. This can be attributed to
the addition of the CRF layer, which exploits inter-token dependencies. The
BERT+BiLSTM+CRF model is much heavier with a larger number of parameters and
overfits the training dataset due to the smaller number of training instances.

For the Chinese language, we observe that both the BERT+CRF (0.8564 F1 score)
and BERT+BiLSTM+CRF (0.8558 F1 score) beat BERT+Linear (0.846 F1 score).

Our models outperform the baseline for both the languages. Our best performing
model BERT+CRF beats the baseline F1 score by around 5$\%$ for Spanish and by
around 10$\%$ for Chinese.

The details of the performance of the BERT+CRF model in the evaluation phase
are shown in Table 8 for the Spanish language and the Table 8 for the Chinese
language. We observe a drop in performance on the test dataset compared to the
performance on the validation dataset. The model scores 0.6120 F1 for Spanish
and 0.6695 F1 for the Chinese language.

##  8 Error Analysis

We perform error analysis on all 3 different models. We qualitatively analyze
the predictions on the validation dataset for both languages. As the final
evaluation test set in blind, we are unable to perform analysis on the same.
We find that the labels GRP (Group), PROD (Product), and CW (Creative Work)
are the most inaccurately predicted labels for the Spanish models. This
conforms to our hypothesis that the long-tailed nature of these entities
(which means the frequency of occurrence of such entity types in the general
literature of the target language is rare). Hence, the model has the most
difficulty in recognizing these entities from the contextual sentences. The
other label types are more common and were present in the CoNLL dataset as
well. We also notice that the BERT+Linear does marginally better than BERT+CRF
on predicting such labels (for the Spanish language), despite it not being the
best performing model overall. This can be attributed to it being a lighter
model, imparting it the capability of generalizing better while training on a
relatively lower amount of training instances. BERT+CRF benefits from having
CRF along with the lower number of parameters compared to the BERT+BiLSTM+CRF
model. This results in it having a better performance compared to both the
other models. The drop in the performance of the model on the blind test
dataset can be attributed to the model not generalizing well to handle
instances of questions and short search queries in the additional test set.

##  9 Conclusion and Future Work

We have introduced strong improvements over the baseline for the shared task
of complex NER for low resource languages. We leverage the Whole Word Masking
objective to obtain a better performance in this low-resource setting. We
perform extensive experiments and find that simple BERT-CRF based models
perform strongly against other heavier models even in such low resource
semantically ambiguous setting as evident by the final evaluation rankings. We
find this approach to give a higher performance as it is able to utilize the
contextual information from a sequence of tokens and learn inter-token
dependencies to accurately predict the named entity labels. We also conduct
qualitative error analysis and describe our findings. For future work, we aim
to leverage these findings to circumvent the label scarcity problem in other
low-resource languages and code mixed data.

##  10 Acknowledgment

We thank the SemEval team, the task organizers, and other participants of this
shared task.

## References

  * Baevski et al. (2019) Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. 2019\.  Cloze-driven pretraining of self-attention networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5360–5369, Hong Kong, China. Association for Computational Linguistics.
  * Bari et al. (2019) M. Saiful Bari, Shafiq R. Joty, and Prathyusha Jwalapuram. 2019.  Zero-resource cross-lingual named entity recognition.  _CoRR_ , abs/1911.09812.
  * Cañete et al. (2020) José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. 2020.  Spanish pre-trained bert model and evaluation data.  In _PML4DC at ICLR 2020_.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _CoRR_ , abs/1911.02116.
  * Cui et al. (2020) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020\.  Revisiting pre-trained models for Chinese natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 657–668, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Feng et al. (2018) Xiaocheng Feng, Xiachong Feng, Bing Qin, Zhangyin Feng, and Ting Liu. 2018.  Improving low resource named entity recognition using cross-lingual knowledge transfer.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , IJCAI’18, page 4071–4077. AAAI Press.
  * Fetahu et al. (2021) Besnik Fetahu, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021.  Gazetteer Enhanced Named Entity Recognition for Code-Mixed Web Queries.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1677–1681.
  * Hedderich et al. (2021) Michael A. Hedderich, Lukas Lange, and Dietrich Klakow. 2021.  ANEA: distant supervision for low-resource named entity recognition.  _CoRR_ , abs/2102.13129.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional LSTM-CRF models for sequence tagging.  _CoRR_ , abs/1508.01991.
  * Jadhav (2020) Swapnil Ashok Jadhav. 2020.  Detecting potential topics in news using bert, CRF and wikipedia.  _CoRR_ , abs/2002.11402.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 260–270, San Diego, California. Association for Computational Linguistics.
  * Luoma and Pyysalo (2020) Jouni Luoma and Sampo Pyysalo. 2020.  Exploring cross-sentence contexts for named entity recognition with BERT.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 904–914, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Malmasi et al. (2022a) Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022a.  MultiCoNER: a Large-scale Multilingual dataset for Complex Named Entity Recognition.
  * Malmasi et al. (2022b) Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022b.  SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER).  In _Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)_. Association for Computational Linguistics.
  * Meng et al. (2021) Tao Meng, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021.  GEMNET: Effective gated gazetteer representations for recognizing complex entities in low-context input.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1499–1512.
  * Rahimi et al. (2019a) Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019a.  Massively multilingual transfer for NER.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 151–164, Florence, Italy. Association for Computational Linguistics.
  * Rahimi et al. (2019b) Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019b.  Multilingual ner transfer for low-resource languages.  _ArXiv_ , abs/1902.00193.
  * Ramshaw and Marcus (1999) Lance A Ramshaw and Mitchell P Marcus. 1999.  Text chunking using transformation-based learning.  In _Natural language processing using very large corpora_ , pages 157–176. Springer.
  * Rijhwani et al. (2020) Shruti Rijhwani, Shuyan Zhou, Graham Neubig, and Jaime Carbonell. 2020.  Soft gazetteers for low-resource named entity recognition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8118–8123, Online. Association for Computational Linguistics.
  * Schweter and Akbik (2020) Stefan Schweter and Alan Akbik. 2020.  FLERT: document-level features for named entity recognition.  _CoRR_ , abs/2011.06993.
  * Wang et al. (2020) Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2020.  Automated concatenation of embeddings for structured prediction.  _CoRR_ , abs/2010.05006.
  * Wang et al. (2021a) Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021a.  Automated concatenation of embeddings for structured prediction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2643–2660, Online. Association for Computational Linguistics.
  * Wang et al. (2021b) Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021b.  Improving named entity recognition by external context retrieving and cooperative learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1800–1812, Online. Association for Computational Linguistics.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _CoRR_ , abs/1609.08144.
  * Xie et al. (2018) Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018\.  Neural cross-lingual named entity recognition with minimal resources.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 369–379, Brussels, Belgium. Association for Computational Linguistics.
  * Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020\.  LUKE: Deep contextualized entity representations with entity-aware self-attention.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6442–6454, Online. Association for Computational Linguistics.
  * Ye et al. (2021) Deming Ye, Yankai Lin, and Maosong Sun. 2021.  Pack together: Entity and relation extraction with levitated marker.  _CoRR_ , abs/2109.06067.
  * Zhou and Chen (2021) Wenxuan Zhou and Muhao Chen. 2021.  Learning from noisy labels for entity-centric information extraction.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5381–5392, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.