# COVID Induced Digital Inequality for Senior Citizens

Nicky Qiu<EMAIL_ADDRESS>
(2018)

###### Abstract.

The global pandemic of COVID-19 has fundamentally changed how people interact,
especially with the introduction of technology-based measures that aim at
curbing the spread of the virus. As the country that currently implements one
of the tightest technology-based COVID prevention policy, China has protected
its citizen with a prolonged peaceful time of zero case as well as a fast
reaction to potential upsurging of the disease. However, such mobile-based
technology does come with sacrifices, especially for senior citizens who find
themselves difficult to adapt to modern technologies. In this study, we
demonstrated the fact that most senior citizens find it difficult to use the
health code apps called “JKM”, to which they responded by cutting down on
travel and reducing local commuting to locations where the verification of JKM
is needed. Such compromise has physical and mental consequences and leads to
inequalities in infrastructure, social isolation and self-sufficiency. As we
illustrated in the paper, such decrease in life quality of senior citizens can
be greatly reduced if improvements on the user interactions of the JKM can be
implemented. To the best of our knowledge, we are the first systemic study of
digital inequality due to mobile-based COVID prevention technologies for
senior citizens in China. As similar technologies become widely adopted around
the world, we wish to shed light on how widened digital inequality
increasingly affects the life quality of senior citizens in the pandemic era.

datasets, neural networks, gaze detection, text tagging

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††conference: Woodstock ’18: ACM Symposium on Neural
Gaze Detection; June 03–05, 2018; Woodstock, NY††booktitle: Woodstock ’18: ACM
Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computer systems organization
Embedded systems††ccs: Computer systems organization Redundancy††ccs: Computer
systems organization Robotics††ccs: Networks Network reliability

##  1\. Introduction

In early 2020, COVID-19 hit the world by surprise, and has quickly evolved
into a global pandemic. Nowadays, although vaccines have been widely
available, the unenforceable national vaccination rates and the possibility of
vaccine penetration makes disease prevention a long term mission for at least
the foreseeable future. As COVID ranks as one of the most transmissible
disease that mankind has ever discovered (Wang and Alexander, 2021), a key
measurements to prevent spreading is contact tracing: locating people who have
had close contact with those infected. Mobile-based technology, with its wide
availability and location tracking capability, is thus chosen to estimate each
individual’s risk of exposure to the disease. As a result, mobile tracking
tools are developed and applied in many countries around the world, including
the United States, the United Kingdom(Wymant et al., 2021), China (Liang,
2020; Zhou et al., 2021; Chen et al., 2021), and Saudi Arabia (Hidayat-ur
Rehman et al., 2021).

Suffered from the first and one of the deadest pandemic, China has adopted a
zero-tolerance disease control with a mobile app-based health codes checking
system: a health code called “JKM” that evaluates the risk of an individual
based on the testing history and contact history to those infected, and a
“trajectory code” that locates the individual and tracks their travel history.
Each province has its own JKM that incorporates the same guidelines from the
central disease control system in addition to other risk factors determined by
the local authorities. Each individual is required to show his/her JKM and
sometimes in combination of trajectory code in order to enter almost all
public area throughout China. Such mobile based technology has been proven
effective as China has many of the recent outbreaks under control.

The widely successful digital-based COVID policy implemented in China,
however, does not come without sacrifice. People who do not use smartphones
are unable to realize location tracking or JKM display, thus unable to enter
most public places. This raises not only physical, but also psychological
concerns in an already stressful event like COVID. In addition, JKM is
developed and integrated into two of the most widely used Chinese mobile
applications: Wechat and Alipay. These two mobile apps, however, are designed
for social networking and mobile payment, respectively, and do not have a
dedicated user interface for JKM. Such design makes it even harder for senior
citizens who, on average, are already not as good at using smartphones. This
will aggravate the already widened digital inequality for senior citizens in
China.

We should not neglect the obstacles that seniors encounter in everyday life
and in travel when using the mandatory JKM, as major countries continue to
experience population aging. In 2020, China had 190.6 million adults aged 65
and over, accounting for 13.5 percent of the total population (Jin et al.,
2022), and this number continues to increase. More usable JKM will benefit a
large portion of the population on a daily basis and reduce digital
inequality.

In this paper, we analyzed the extend to which senior citizens’ everyday lives
are affected by the JKM app in China. Our results showed that both their
travel intentions and their willingness to commute to local activity centers
and grocery stores are significantly reduced. This illustrated senior
citizens’ adaptive behavior to the complexity of the interactive process of
the JKM. Further analysis indicated that such phenomenon had consequently
reduced the sense of social acceptance, confidence, and self-sufficiency of
senior citizens in China, making them more and more reliable on their
children, who can usually operate the JKM more easily. Moreover, as the
trajectory code records a person’s travel history and as senior citizen
usually relies on others to show their JKM, privacy concerns arise which
further prevents them from traveling.

Fortunately, our analysis showed that most frustrations of the senior citizens
on using the JKM app can be alleviated by a better user experience design. For
example, our survey indicated that a dedicated JKM app that does not reside in
any of the exiting apps would significantly improve the user experience.
Others believe that user interface improvements such as large fonts and better
layout will also help. In addition, alternative ways to acquire the health
verification such as phone calls and sms will also significantly improve the
experience of senior citizens.

##  2\. Backgrounds

We review some of the background to this work, ranging from digital inequality
induced from the adaptation of the technology driven society to the potential
impacts that COVID have brought to for senior citizens. We also review a list
of mobile based contact tracing applications across the world.

###  2.1. Digital Inequality

Technology especially those aimed at facilitate communication such as Internet
and mobile devices, has made it significantly easier for people to connect and
the interact. As the society transforms to be more and more relied on those
technology, people who are more adapted to take advantage of these tools
became more privileged in this setting over those who are not, creating what
we called digital inequality(DiMaggio et al., 2001; Robinson, 2009; DiMaggio
et al., 2004b; Kvasny, 2006). Seniors (Friemel, 2016), low income
families(Hsieh et al., 2008), and minorities (d’Haenens and Ogan, 2013; Katz
et al., 2017) are more likely to be affected as their access to the technology
and the way how they use it is often limited compare to other groups (DiMaggio
et al., 2004a). As the majority of the major countries entering a decline in
fertility rate and consequently a rapid growth of the aging population,
digital divide for the aged is becoming a pressing societal issue. Older
adults are highly heterogeneous in terms of technology use (van Boekel et al.,
2017), which are mostly impacted by their prior employment experiences.
Research studies have showed that old adults tend to develop their own
routines in using technology (Quan-Haase et al., 2016), which suggested the
need of interactive designs specifically targeted at old adults.

###  2.2. Pandemic Impacts on Technology Use

The COVID-19 pandemic have fundamentally shifted how people use technology. As
in-person communications being significantly reduced, either as a personal
precaution or as a government enforced disease prevention policy, more and
more interactive experiences are moved from offline to online. The use of
technology will significantly affect the quality of the interactions ranging
from online learning (Adnan and Anwar, 2020; Chakraborty et al., 2021; Liguori
and Winkler, 2020), information seeking (Reisdorf et al., 2021), health care
(Okonkwo et al., 2021), work (Alaqra and Kitkowska, 2021), to communications
(Alaqra and Kitkowska, 2021). As we get more isolated compared to prior
pandemic, people seek help from online to deal with COVID-19 Loneliness (Niu
et al., 2021). Privacy concerns raise as more interactions are exposed to the
digital tools than ever (Niu et al., 2021; Seberger and Patil, 2021). And the
pandemic seem to impacts the old adults more than other age groups (Van
Tilburg et al., 2021). Needless to say, the unique challenges of using
technologies in the pandemic era call for changes in designing technology
interface for better access.

###  2.3. Mobile Based Contact Tracing

A key approach in curbing the COVID pandemic is the quarantine of the infected
individuals along their close contacts. Smart phone devices, due to their wide
availability and their GPS tracing capabilities, become an ideal tool to
conduct contact tracing. Various App based contact tracing tools are developed
(Kondylakis et al., 2020; Adeniyi et al., 2020). Many of them are developed
officially by the local authorities include the JKM System by the Chinese
Government (Liang, 2020; Zhou et al., 2021; Chen et al., 2021), Tawakkalna by
Saudi Arabia (Hidayat-ur Rehman et al., 2021), NHS by the UK government
(Wymant et al., 2021), and COVIDSafe by Australia (Yang et al., 2021). As with
any other technologies, old adults often found themselves difficult to adopt
to it compared to other age groups. In countries (e.g, China) where mobile
based apps are required to access most of the public spaces, the access to
those places from senior citizens are also greatly constrained. As a results,
many research calls for studies of impacts of such technology to old adults
(Wang and Jia, 2021) and many early results have find preliminary evidences
(Alharbi et al., 2021; Zhu and Pan, 2021) on the pain points seniors
experience when interacting with these technologies.

##  3\. JKM System Revisited

The Chinese JKM is a complex system consists of a centralized database and two
set of QR codes: JKM and Trajectory Code, as illustrated in Fig. 1. Those two
QR codes are used in combination to determine admission to public facilities
as well as the need for quarantine.

Figure 1. Example of the local JKM (left) and the trajectory code (right).

###  3.1. JKM

Figure 2. Questionnaire for Acquiring JKM for a Different Province  Table 1. Major Survey Questions Question | Answer
---|---
* Are you satisfied with JKM’s design? | Rate on scale of 1 to 5
How to make JKM easier to use? | A. Enlarger text; B. Make JKM an individual APP;
| C. Text or call $\\#12345$ to receive JKM status; D. Other
If COVID risk rating is low, what is your willingness to go |
to essential locations (supermarket/bus/hospitals…)? |
When JKM is not checked | Rate on scale of 1 to 5
When JKM is checked | Rate on scale of 1 to 5
When there is a more convenience way of checking JKM, |
or someone helps you find your JKM | Rate on scale of 1 to 5
If COVID risk rating is low, what is your willingness to go to |
leisure locations (friends places/malls/dances/extracurriculums…)? |
When JKM is not checked | Rate on scale of 1 to 5
When JKM is checked | Rate on scale of 1 to 5
When there is a more convenience way of checking JKM, |
or someone helps you find your JKM | Rate on scale of 1 to 5
If COVID risk rating is low, what is your willingness |
to travel to other cities? |
When JKM is not checked | Rate on scale of 1 to 5
When JKM is checked | Rate on scale of 1 to 5
When there is a more convenience way of checking JKM, |
or someone helps you find your JKM | Rate on scale of 1 to 5
Please rate your proficiency in phone operation | Rate on scale of 1 to 5
What are the 3 APP’s that you use the most? | [open ended]

The JKM provides a single entry to illustrate an individual’s health condition
related to COVID based on information in the central government’s database.
There are three status of the JKM: Green, yellow, and red, indicating
different level of risks for that specific individual. Information will be
used when JKM is determined includes: 1) COVID testing results (if any), 2)
vaccine status, and 3) whether a person is considered close contact to an
infected individual based on data from trajectory code (discussed in the next
section), and 4)other situations specified by the local or central government
(e.g., when mandatory COVID testing is enforced JKM from anyone who did not
conduct the test will be overridden to be red and will not resume until the
results are negative). One unique characteristic for the Chinese JKM is that
each province will have its own standard in determining the status of the JKM
based on the same set of input data, as illustrated in Fig.1. This mechanics
provides flexibility for the provinces to set their own disease prevention
rules. When traveling to a different province, a questionnaire will have to be
filled for a set of additional questions in order to register for the JKM
belongs to a different states. An example has been provided in Fig. 2. Many of
the questions are multiple choice ones asking the exact travel destination as
well as the health condition and the the contact history within the past 14
days. One will have to fill out the questionnaire each time when he/she visits
a new province regardless of whether that province has been visited before. In
our analysis in the later part of the paper, the registration procedure set a
major obstacle to senior citizens as fonts in the phones are usually too small
for most to recognized and the survey is usually filed in the boarder of the
new province where lots of travellers waiting in line.

###  3.2. Trajectory Code

Figure 3. Verification Process for Trajectory Code

The trajectory has dual functions: 1) it exists as a data collection mechanism
to acquire user’s geo-spatial locations, and 2) it can be used to illustrate
an individual’s health status similar to the JKM. As the GPS location can
often be forged by providing software simulated locations, the Chinese
trajectory code uses cellular signal to locate an individual’s location. In
order to acquire the trajectory code, one will have to enter his/her Cheese
mobile phone number in order to receive an verification pin code, which
interface is illustrated in Fig. 3. The geo-location will be inferred using
cellular services which is usually much different to forge. Once the
trajectory code is requested, one will have to verify it through sms messages,
in which case the system will collect a data point of presence and will be
used later in constructing the actual travel trajectory. Such a practice might
be very common for young adults who are constantly using online services that
require phone verification. However, for senior citizens this procedure can be
difficult to achieve as they will have to fill in the phone number, click on
“authorizing to release the information” checkbox and request and fill in
their verification code. In our interview, we find many senior citizens
difficult to locate and the understand the content on the page due to the
relatively small font size and the fact that most old adults have presbyopia,
which prevented them from viewing close items clearly.

###  3.3. Software Entry Points

JKM can be accessed through either Wechat, Alipay, or a service app provided
from the government of each province. JKM is developed as a service or a
function within each of the platform, and the entry point of each platform
varies. For wechat, which is the most widely used apps for accessing JKM in
our experience, is a social networking apps. In order to access JKM, one will
have to either scan the QR code or to search for the name under the in-app
developed program section. In Alipay, which is a payment app, one can access
to the JKM in a similar way but the search procedure is different. In the case
of the official apps provided by the government, it is developed as a service
of that app and access methods vary for each province. As we will later show
in our analysis, the lack of independent app is one of the main complains from
senior citizens. Improving the interactive experience of the users will
significantly improve senior citizens’ will to travel and to perform their
day-to-day job.

###  3.4. Facility Admission and Mandatory Quarantine

JKM is used to determine the access to public facilities and the risk of
mandatory quarantine in China. Only the green code will guarantee access to
public facilities. This means that JKM will be checked upon every visit to
public facilities such as building, markets, transportation, hospitals,
schools, and cinemas. JKM is usually the primary JKM to check. In case where
the local disease becomes a bit severe, trajectory code will also needed to be
checked. Mandatory quarantine will be conducted based on the local authority’s
assessments of individual risks, usually when the travel trajectory insects
with those who have been infected.

##  4\. Methods

We conducted surveys and interviews among senior citizens in Shanghai to
understand the impact JKM has made on the aspects of their lives.

###  4.1. Data Collection

We recruited participants from 5 different locations in Shanghai, China:
Shanghai Old-Age University, Xiangyang Park, Jiangpu Park, Taichi class, and
Golden Harbor Apartment. We picked these locations because they are spread out
across Shanghai, and they are where seniors with various backgrounds come and
gather.

####  4.1.1. Survey

.

We designed the survey to make it accessible to our target audience in order
to minimize the systematic bias. The survey is made in paper format and
distributed in person, therefore seniors who are unable to fill out online
surveys are not discouraged or excluded from the population. The fonts and
color contrast are increase so as to make it more readable for vision
degenerated seniors. For seniors who are still not able to read due to vision
or illiteracy, the survey is read to them. To hear the voice of seniors with
various academic backgrounds, we phrased the questions in everyday terms and
minimized barriers of academic ability such as ability to read charts. The
survey takes about 15 to 20 minutes to complete. We demonstrated some of the
major survey questions in Tab (tab:survey).

####  4.1.2. Interview

.

We conducted interviews in the same 5 locations as survey with 11 seniors who
are willing to share their opinions. We observed organic conversations among
participants and encouraged participants to liberally think out loud. Instead
of rigorously structured Q & A, we only ask questions occasionally to
facilitate the discussion and have our key questions answered. The answers are
not recorded due to participants’ concerns of privacy, but are transcribed for
research purposes. The interviews last from 20 minutes to 50 minutes. We have
summarized the list of interview questions in Tab. 2.

Table 2. Interview Questions We Asked On-site. *What kind of JKM do you use?
---
(if participant does not use JKM)
how do you go to places which checks JKM, such as the hospital?
What is the impact of JKM on your willingness to travel?
How often do you use your cell phone?
What APPs do you use most often?

###  4.2. Dataset Overview

Table 3. Demographics Statistics of Our Survey Data Gender | Age | Education | Platform | Live with?
---|---|---|---|---
Female | 62% | 55 - 59 | 23% | Under Primary School | 8% | IOS | 54% | Self | 8%
Male | 38% | 60 - 69 | 46% | Primary School | 8% | Android | 15% | Partner | 62%
|  | 70 - 79 | 8% | Middle School | 23% | Others | 31% | Children | 23%
|  | 80 - 89 | 23% | High School | 54% |  |  | Grand children and children | 8%

Our sample was diverse in terms of demographics (see Table 3 for key
demographics). The 8 female and 5 male participants ranged in age from 58 to
83, living by themselves, with partners, children, or grand children. They
have educational backgrounds ranging from under primary school to college, and
occupations including accountants, government officials, professors, workers,
servers and so forth. About three quarters of them use a smartphone, such as
Apple, Huawei or others. At the time of our survey, two-thirds of our
participants uses JKM through an APP, and others find an alternative way when
JKM is checked.

##  5\. Shifts in Travel Patterns by JKM

Senior citizens’ travel patterns changed significantly in the era of COVID.
Part of this is inevitable as seniors are at greater risks of developing
severe symptoms of COVID compared to other age groups. Another part of the
story has to do with the design of the JKM, which created an extra barrier for
seniors to travel. In this section, we analyze the results from survey and
interview data and study how the JKM itself other than the COVID affects the
travel patterns of the Seniors.

###  5.1. Survey Results

Figure 4. Changes in Travel Patterns with JKM Required and JKM not Required,
Under Low Risk Level, to a) Essential Places, b) Leisure Places, and c) Other
Cities

We analyzed the results from the survey on how much JKM changed senior
citizens’ willingness of traveling. This survey assumes that all travel
destination is classified as “low risk” 111defined as an area that has
accumulative COVID cases of 50 or below in the last 14 days.  by the Chinese
government, as indicated in our survey questions in Tab 1. This assumption of
made sure that the changes in travel intentions are not caused by COVID
itself. We categories the destinations into three distinct types: a) essential
places (e.g. supermarkets, hospitals, subway stations etc), b) entertainment
(e.g. friend’s places, malls, parks, extracurricular classes etc) and c)
tourism, which destination usually lies in another city.

We compared the data with/without checking JKM, and the results are
illustrated in Fig. 4. Here we see that travel intentions for each category
falls into 0-5, and a reference line can be drawn with $x=y$ if no travel
pattern changes can be recognized. Any points above $x=y$ means that the
checking JKM makes senior citizens more willing to travel to a destination. On
the other hand, data points below $x=y$ means that checking JKM creates extra
hurdles to prevent old adults from going to these locations. Overall,
participants’ desire to travel decreases when JKM is checked versus not
checked, with everything else held equal. Out of the three hypothetical
situation, the desire to travel to essential places such as supermarkets and
hospitals are least impacted. On the one hand, like many participants said
“you have to go, you have to go”. On the other hand, seniors still choose to
decrease their frequency of travel when JKM is checked. Fig. 4 also shows that
places for entertainment activities, such as friend’s places, malls, parks,
extracurricular classes etc are most impacted. This is because participants
consider these places as “do not have to go”, thus would choose to go less
often when these places are less accessible. Although it does not hurt their
basic living and health needs the way essential locations do, it does
contribute to their social isolation and mental well being in ways that are
less noticeable.

###  5.2. Reduced Access to Healthcare

In our interview, we also evaluated the degree to which the respondents’
access to hospitals and healthcare services are affected. One participant said
she does go to the hospital “as long as I could ensure [the symptom] is in
check, because going to the hospital and checking the JKM is too much hassle”.
The hospital is only 10 minutes walk from her home, yet checking of JKM and
digital appointment is too much of a mental burden for her. Such behavior
could lead to serious consequences as treatments could be delayed and her
health can be negatively impacted by various degrees.

###  5.3. Feeling of Denial by the Society

Even in cases when seniors do visit the hospital, most of them have to seek
assistance when JKM is checked, and in cases when no help is available, they
may still give up going to their destination and return home. As described by
a participant “[the receptionist at hospital] said no JKM no access, so I
might as well not go…in the end one doctor stepped in and helped me [display
JKM]”. This participant described the recipient as “evil”, and this feeling of
opposition is not uncommon. Many seniors expressed feelings of being
unaccepted and unwanted by the society as they are unable to follow the rules
of the society and further, not even considered when these rules were made in
the first place.

“[some younger people] consider us as burdens to the society, that they have
to pay tax to feed us. They hope us better die. They don’t know that one day
they will age too.”

###  5.4. Increased Dependencies on Others

As JKM increases seniors’ level of dependency on others, it also consumes
their level of confidence. Some participants ask their children to find and
screenshot their JKM for them. Sometimes a recent screenshot of JKM is
accepted, so the participants will have to ask their children to do this for
them often. Sometimes a screenshot is not accepted, then the participants
would have to turn to the staff for help. For these participants, a feeling of
insecurity is always around them, for they do not know when their screenshot
will be rejected. For participants who want to learn how to find JKM, it is
not much easier, either. “[my child] showed me three times how to find [JKM]
but I still can not remember how to do it. Asking [my child] one or two times
is fine; after three or four times they must be annoyed” said a participant.
This concern about the relationship with children and self-awareness is not
uncommon. Not being able to use an unfamiliar technology which is officially
widely required for travel does hurt some seniors’ level of confidence.

##  6\. Senior-unfriendly Designs of JKM

Recognizing that the JKM created an extract barrier for seniors’ travel
intentions, we now seek to understand the set of design flaws that of the
current JKM. We first analysis the relationship between respondents’ personal
assessments of their familiarity of smartphones and its relations to their
perceived usability of JKM’s UI. We then zoom into several common obstacles
that prevents the elder individuals from using the JKM in a convenient way.

###  6.1. Smartphones Knowledge and JKM Satisfactory

Figure 5. The respondents’ self-assessed smartphone proficiency (x axis) and
their perceived satisfactory on JKM’s UI (y axis)

We first want to figure out whether the frustrations about the JKM UI is only
restricted within the group of senior citizens who are have little knowledge
about smartphones in general. In Fig. 5, we plotted the self-assessed
smartphone knowledge in the x axis along with UI satisfactory (y axis). Here
we do see a correlation betwen the general satisfactory and the skillfulness
of the users. However, in many situation even participants who are tech savvy
find JKM hard to use. Fig. 5 shows that about 80$\%$ of participants with
various familiarity with smartphones perceive the UI of JKM as very hard to
use.

###  6.2. Commonly Used Apps Among Respondents

Figure 6. Top APPs Used by Respondents

We also analyzed the most commonly used apps among the respondents. The most
commonly used APPs among the participants are WeChat (50$\%$, social media),
Phone (37.5$\%$, essential phone call app), TikTok (25$\%$, short video) and
Ali Pay (25$\%$, mobile payment). As JKM is embedded into WeChat and Alipay,
this does means that the majority of the users ($75\%$) are already familiar
with the entry points of the main app that hosts the JKM. However, even that’s
the case respondents still find them difficult to use, probably because the
usecase of the JKM deviates from the main day-to-day applications of these
apps. And even seniors who have already had experiences with those apps still
need to learn how to operate JKJ with those platforms.

###  6.3. Accessibility and Availability of JKM

Other aspects that make using JKM challenging for seniors include complex
entrance, inaccessibility without internet, and frequent updates. JKM’s
entrance is within WeChat or AliPay, whose main function is social networking
and mobile payment. JKM is listed as an icon, along with many other features,
within WeChat mini programs/front page of AliPay (screenshots). This makes it
hard for the participants to find the JKM and contributes to their learning
curve - “[my child] showed me three times how to find [JKM] but I still do not
remember how to do it” said a participant. Several participants also expressed
frustrations about not being able to load JKM when there is low cellular
signal. They have had experiences when they were blocked at the entrance of
their destination not because they did not know how to show their JKM, but
because of poor internet. Another commonly mentioned frustration is the
frequency of the updates. JKM expires in several days and when it does, one
has to manually update it by sending a text to their phone containing a
verification code and then entering it in JKM. Users usually will not know
when this update needs to take place until when they need to present their
JKM, so this lengthy process can slow down their travel unexpectedly. In
addition, a participant expressed concern in not being able to enter the
verification code in time which is only valid for 3 minutes.

##  7\. Improvement Ideas for JKM

After we understand why the current design of JKM makes seniors frustrated, we
now seek to explore ideas to improve it. We first analyze the improvement
ideas based on our survey data. We will now show how these improvements can
really have an impacts on seniors’ willingness to travel.

###  7.1. Most Requested Features

Figure 7. Top requested features by our senior respondents

Figure 8. Changes in Travel Patterns with Current Version of JKM vs Improved
JKM, Under Low Risk Level, to a) Essential Places, b) Leisure Places, and c)
Other Cities

We analyzed the top requested features based on our survey data. Fig. 7 shows
a summary of the results identified by the survey participants.

####  7.1.1. JKM as A Standalone APP

Many participants mentioned that seniors would like JKM to be as simple as
possible. Therefore, the number one requested feature is to make JKM an
individual APP, minimizing the steps it takes to find the JKM. This is opposed
to the current procedure of retrieving it from Wechat or Alipay, which are two
apps that are designed for other purposes. Especially for senior citizens who
have presbyopia, making it as a standalone app will eliminate the troubles of
finding it within the other platforms.

####  7.1.2. Receives JKM Status via Text or Call

Anther top requested feature from senior citizens is to retrieve JKM using an
more traditional approach such as sms or phone call. This method elaborates on
seniors’ existing knowledge on the phone. Cell phones are produced decades
before smart phones are, and telephones have an even longer history. As a
result, a large number of seniors are familiar with texting and most of them
are able to make phone calls. Therefore, it would be easy for seniors to be
able to receive their JKM status in an alternative way, for example to text or
call a certain number. This would benefit people with visual impairment as
well.

####  7.1.3. Larger Text on the UI

36$\%$ of the participants mentioned larger text on the UI as an improvement,
indicating the current texts are not big enough for seniors. As vision
declines over years, larger text can help seniors use JKM more easily.

###  7.2. Projected Improvements on Travel Patterns

As we make improvements on the interactive design of the JKM, we seek to
estimate its potential impacts on seniors’ travel patterns by asking the same
question by with the hypothetical setting of UI being improved. Fig. 8 shows
that if JKM is better designed, participants’ desire to travel to essential
places, leisure places and other cities all increase, when all else are equal
and COVID risk level is low.

###  7.3. Society Acceptance

More usable JKM also allows seniors use JKM more easily which helps improve
their confidence level and self sufficiency. Two participants that we
interviewed displayed high level of confidence and independence. They live by
themselves - “why would I live with my children? That would be so annoying [to
me]” “I have two laptops and two phones. I can spend time with them all day
long”. Being tech savvy, they are able to amuse themselves with the internet,
watch news online and buy stocks on the phone. They feel a close connection to
the society. They also do not feel the rush to find a partner to “take care of
them”, like many seniors do in China. “I need a man who attracts me
spiritually, but I have not find him yet.”

##  8\. Discussions

There are 5 levels of human needs according to Maslow’s Hierarchy of
Needs(McLeod, 2007). Unfortunately, the imposition of JKM as a prerequisite
for entry into most public places is hurting the needs of Chinese senior
citizens. Some seniors suppress their physiological needs with less visits to
essential places like the hospital. Some seniors have unfulfilled
belongingness and love needs as they lose the connection to the society or
their very own children. Finally, some seniors feels less esteem and self-
actualization as they are unable to understand how to use JKM, nor can they
travel to the places by themselves like before without having to depend on
others for help.

One important limitation of this study is the participation of only seniors in
China, who may not be representative of other countries. More case studies in
other countries would contribute to the understanding of such inequality in a
different social context. Second, users’ self-reported smartphone proficiency
may be subjective: in the future, objective tests can be used to evaluate
participants’ proficiency. Third, a portion of the interviews were conducted
in group format, which may lead to bias such as group think - tendency to
agree with others in a group setting. Future interviews can be conducted in
private before in groups to avoid group think. Finally, when asking
participants about their opinions on a better-designed JKM, we did not show
participants a redesigned App with improved user experience. Future works can
include prototypes and A/B testing which can help participants visualize the
improvement ideas and evaluate them more intuitively.

##  9\. Conclusions

In this paper, we studied senior citizens’ experience and opinions on JKM, a
code which every citizen is required to show upon entry to most public places
in China, such as hospital, malls and train stations. We revealed the
inequalities induced by technology during the Covid pandemic: access to social
infrastructure, social isolation, and self-sufficiency. Reflecting on the
feedback from the seniors citizens, we explored possible design improvements
to the JKM and the positive impact.

## References

  * (1)
  * Adeniyi et al. (2020) Emmanuel Abidemi Adeniyi, Joseph Bamidele Awotunde, Roseline Oluwaseun Ogundokun, Paul Oluwatoba Kolawole, Moses Kazeem Abiodun, and Abosede Adepeju Adeniyi. 2020\.  Mobile health application and COVID-19: Opportunities and challenges.  _Journal of Critical Reviews_ 7, 15 (2020), 3481–3488.
  * Adnan and Anwar (2020) Muhammad Adnan and Kainat Anwar. 2020.  Online Learning amid the COVID-19 Pandemic: Students’ Perspectives.  _Online Submission_ 2, 1 (2020), 45–51.
  * Alaqra and Kitkowska (2021) Ala Sarah Alaqra and Agnieszka Kitkowska. 2021.  Impact of Intrinsic Factors and COVID-19 Pandemic on the Use of Technology. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–7.
  * Alharbi et al. (2021) Raghad A Alharbi, Faisal T Altayyari, Farah S Alamri, and Sultan A Alharthi. 2021.  Pandemic-Driven Technology During COVID-19: Experiences of Older Adults. In _Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing_. 5–9.
  * Chakraborty et al. (2021) Pinaki Chakraborty, Prabhat Mittal, Manu Sheel Gupta, Savita Yadav, and Anshika Arora. 2021\.  Opinion of students on online education during the COVID-19 pandemic.  _Human Behavior and Emerging Technologies_ 3, 3 (2021), 357–365.
  * Chen et al. (2021) Shan Chen, Tianjiao Liu, Xin Li, Yingjuan Luo, Li Xiao, Libing Zhang, Rongkang Wen, and Yonghong Lin. 2021.  Health QR Code Application in the Novel Containment Strategy and Healthcare Plan for Pregnant Women and Children Under Quarantine During the Summer Outbreak of SARS-CoV-2 Delta Variant in Chengdu, China: An Observational Study.  _Risk Management and Healthcare Policy_ 14 (2021), 4499–4510.
  * DiMaggio et al. (2001) Paul DiMaggio, Eszter Hargittai, et al. 2001\.  From the ‘digital divide’to ‘digital inequality’: Studying Internet use as penetration increases.  _Princeton: Center for Arts and Cultural Policy Studies, Woodrow Wilson School, Princeton University_ 4, 1 (2001), 4–2.
  * DiMaggio et al. (2004a) Paul DiMaggio, Eszter Hargittai, Coral Celeste, and Steven Shafer. 2004a.  Digital inequality: From unequal access to differentiated use.  _Social inequality_ (2004), 355–400.
  * DiMaggio et al. (2004b) Paul DiMaggio, Eszter Hargittai, Coral Celeste, Steven Shafer, et al. 2004b.  From unequal access to differentiated use: A literature review and agenda for research on digital inequality.  _Social inequality_ 1 (2004), 355–400.
  * d’Haenens and Ogan (2013) Leen d’Haenens and Christine Ogan. 2013.  Internet-using children and digital inequality: A comparison between majority and minority Europeans.  _Communications-The European Journal of Communication Research_ 38, 1 (2013), 41–60.
  * Friemel (2016) Thomas N Friemel. 2016\.  The digital divide has grown old: Determinants of a digital divide among seniors.  _New media & society_ 18, 2 (2016), 313–331.
  * Hidayat-ur Rehman et al. (2021) Imdadullah Hidayat-ur Rehman, Arshad Ahmad, Manzoor Ahmed, and Aftab Alam. 2021.  Mobile Applications to Fight against COVID-19 Pandemic: The Case of Saudi Arabia.  _Tem Journal-Technology Education Management Informatics_ (2021), 69–77.
  * Hsieh et al. (2008) JJ Po-An Hsieh, Arun Rai, and Mark Keil. 2008.  Understanding digital inequality: Comparing continued use behavioral models of the socio-economically advantaged and disadvantaged.  _MIS quarterly_ (2008), 97–126.
  * Jin et al. (2022) Shuyi Jin, Chenxi Li, Xingqi Cao, Chen Chen, Zhihong Ye, and Zuyun Liu. 2022\.  Association of lifestyle with mortality and the mediating role of aging among older adults in China.  _Archives of Gerontology and Geriatrics_ 98 (2022), 104559.
  * Katz et al. (2017) Vikki S Katz, Carmen Gonzalez, and Kevin Clark. 2017\.  Digital inequality and developmental trajectories of low-income, immigrant, and minority children.  _Pediatrics_ 140, Supplement 2 (2017), S132–S136.
  * Kondylakis et al. (2020) Haridimos Kondylakis, Dimitrios G Katehakis, Angelina Kouroubali, Fokion Logothetidis, Andreas Triantafyllidis, Ilias Kalamaras, Konstantinos Votis, and Dimitrios Tzovaras. 2020.  COVID-19 mobile apps: a systematic review of the literature.  _Journal of medical Internet research_ 22, 12 (2020), e23170.
  * Kvasny (2006) Lynette Kvasny. 2006\.  Cultural (re) production of digital inequality in a US community technology initiative.  _Information, Communication & Society_ 9, 02 (2006), 160–181.
  * Liang (2020) Fan Liang. 2020\.  ¡? covid19?¿ COVID-19 and Health Code: How Digital Platforms Tackle the Pandemic in China.  _Social Media+ Society_ 6, 3 (2020), 2056305120947657\.
  * Liguori and Winkler (2020) Eric Liguori and Christoph Winkler. 2020.  From offline to online: Challenges and opportunities for entrepreneurship education following the COVID-19 pandemic.
  * McLeod (2007) Saul McLeod. 2007\.  Maslow’s hierarchy of needs.  _Simply psychology_ 1, 1-18 (2007).
  * Niu et al. (2021) Shuo Niu, Ava Bartolome, Cat Mai, and Nguyen Binh Ha. 2021\.  # StayHome# WithMe: How Do YouTubers Help with COVID-19 Loneliness?. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Okonkwo et al. (2021) Nneoma E Okonkwo, Ugochi T Aguwa, Minyoung Jang, Iman A Barré, Kathleen R Page, Patrick S Sullivan, Chris Beyrer, and Stefan Baral. 2021\.  COVID-19 and the US response: accelerating health inequities.  _BMJ evidence-based medicine_ 26, 4 (2021), 176–179.
  * Quan-Haase et al. (2016) Anabel Quan-Haase, Kim Martin, and Kathleen Schreurs. 2016\.  Interviews with digital seniors: ICT use in the context of everyday life.  _Information, Communication & Society_ 19, 5 (2016), 691–707.
  * Reisdorf et al. (2021) Bibi Reisdorf, Grant Blank, Johannes M Bauer, Shelia Ree Cotten, Craig Robertson, and Megan Knittel. 2021.  Information-Seeking Patterns and COVID-19 in the United States.  _Journal of Quantitative Description: Digital Media_ 1 (2021), 1–38.
  * Robinson (2009) Laura Robinson. 2009\.  A taste for the necessary: A Bourdieuian approach to digital inequality.  _Information, Communication & Society_ 12, 4 (2009), 488–507.
  * Seberger and Patil (2021) John S Seberger and Sameer Patil. 2021.  Us and Them (and It): Social Orientation, Privacy Concerns, and Expected Use of Pandemic-Tracking Apps in the United States. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–19.
  * van Boekel et al. (2017) Leonieke C van Boekel, Sebastiaan TM Peek, and Katrien G Luijkx. 2017.  Diversity in older adults’ use of the internet: identifying subgroups through latent class analysis.  _Journal of medical Internet research_ 19, 5 (2017), e180.
  * Van Tilburg et al. (2021) Theo G Van Tilburg, Stephanie Steinmetz, Elske Stolte, Henriëtte van der Roest, and Daniel H de Vries. 2021.  Loneliness and mental health during the COVID-19 pandemic: A study among Dutch older adults.  _The Journals of Gerontology: Series B_ 76, 7 (2021), e249–e255.
  * Wang and Alexander (2021) Lidong Wang and Cheryl Ann Alexander. 2021.  COVID-19 Compared with Other Viral Diseases: Novelties, Progress, and Challenges.  _Electronic Journal of General Medicine_ 18, 1 (2021).
  * Wang and Jia (2021) Tiantian Wang and Fei Jia. 2021.  The impact of health QR code system on older people in China during the COVID-19 outbreak.  _Age and Ageing_ 50, 1 (2021), 55–56.
  * Wymant et al. (2021) Chris Wymant, Luca Ferretti, Daphne Tsallis, Marcos Charalambides, Lucie Abeler-Dörner, David Bonsall, Robert Hinch, Michelle Kendall, Luke Milsom, Matthew Ayres, et al. 2021\.  The epidemiological impact of the NHS COVID-19 App.  _Nature_ 594, 7863 (2021), 408–412.
  * Yang et al. (2021) Fan Yang, Luke Heemsbergen, and Robbie Fordyce. 2021\.  Comparative analysis of China’s Health Code, Australia’s COVIDSafe and New Zealand’s COVID Tracer Surveillance Apps: a new corona of public health governmentality?  _Media International Australia_ 178, 1 (2021), 182–197.
  * Zhou et al. (2021) Sophia L Zhou, Xianhan Jia, Samuel P Skinner, William Yang, and Isabelle Claude. 2021.  Lessons on mobile apps for COVID-19 from China.  _Journal of Safety Science and Resilience_ 2, 2 (2021), 40–49.
  * Zhu and Pan (2021) Qiming Zhu and Young-Hwan Pan. 2021.  A user interface design of the health QR code system in China.  _Journal of the Korea Convergence Society_ 12, 1 (2021), 37–47.


# Multi-task Pairwise Neural Ranking for Hashtag Segmentation

Mounica Maddela1, Wei Xu1, Daniel Preoţiuc-Pietro2
1 Department of Computer Science and Engineering, The Ohio State University
2 Bloomberg LP
{maddela.4<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Hashtags are often employed on social media and beyond to add metadata to a
textual utterance with the goal of increasing discoverability, aiding search,
or providing additional semantics. However, the semantic content of hashtags
is not straightforward to infer as these represent ad-hoc conventions which
frequently include multiple words joined together and can include
abbreviations and unorthodox spellings. We build a dataset of 12,594 hashtags
split into individual segments and propose a set of approaches for hashtag
segmentation by framing it as a pairwise ranking problem between candidate
segmentations.111Our toolkit along with the code and data are publicly
available at https://github.com/mounicam/hashtag_master Our novel neural
approaches demonstrate 24.6% error reduction in hashtag segmentation accuracy
compared to the current state-of-the-art method. Finally, we demonstrate that
a deeper understanding of hashtag semantics obtained through segmentation is
useful for downstream applications such as sentiment analysis, for which we
achieved a 2.6% increase in average recall on the SemEval 2017 sentiment
analysis dataset.

##  1 Introduction

A hashtag is a keyphrase represented as a sequence of alphanumeric characters
plus underscore, preceded by the # symbol. Hashtags play a central role in
online communication by providing a tool to categorize the millions of posts
generated daily on Twitter, Instagram, etc. They are useful in search,
tracking content about a certain topic Berardi et al. (2011); Ozdikis et al.
(2012), or discovering emerging trends Sampson et al. (2016).

Hashtags often carry very important information, such as emotion Abdul-Mageed
and Ungar (2017), sentiment Mohammad et al. (2013), sarcasm Bamman and Smith
(2015), and named entities Finin et al. (2010); Ritter et al. (2011). However,
inferring the semantics of hashtags is non-trivial since many hashtags contain
multiple tokens joined together, which frequently leads to multiple potential
interpretations (e.g., lion head vs. lionhead). Table 1 shows several examples
of single- and multi-token hashtags. While most hashtags represent a mix of
standard tokens, named entities and event names are prevalent and pose
challenges to both human and automatic comprehension, as these are more likely
to be rare tokens. Hashtags also tend to be shorter to allow fast typing, to
attract attention or to satisfy length limitations imposed by some social
media platforms. Thus, they tend to contain a large number of abbreviations or
non-standard spelling variations (e.g., #iloveu4eva) Han and Baldwin (2011);
Eisenstein (2013), which hinders their understanding.

Type | Single-token |  | Multi-token
---|---|---|---
Named-entity (33.0%) | #lionhead |  | #toyotaprius
Events (14.8%) | #oscars |  | #ipv6summit
Standard (43.6%) | #snowfall |  | #epicfall
Non-standard (11.2%) | #sayin |  | #iloveu4eva
Table 1: Examples of single- (47.1%) and multi-word hashtags (52.9%) and their
categorizations based on a sample of our data.

The goal of our study is to build efficient methods for automatically
splitting a hashtag into a meaningful word sequence. Our contributions are:

  * •

A larger and better curated dataset for this task;

  * •

Framing the problem as pairwise ranking using novel neural approaches, in
contrast to previous work which ignored the relative order of candidate
segmentations;

  * •

A multi-task learning method that uses different sets of features to handle
different types of hashtags;

  * •

Experiments demonstrating that hashtag segmentation improves sentiment
analysis on a benchmark dataset.

Our new dataset includes segmentation for 12,594 unique hashtags and their
associated tweets annotated in a multi-step process for higher quality than
the previous dataset of 1,108 hashtags Bansal et al. (2015). We frame the
segmentation task as a pairwise ranking problem, given a set of candidate
segmentations. We build several neural architectures using this problem
formulation which use corpus-based, linguistic and thesaurus based features.
We further propose a multi-task learning approach which jointly learns segment
ranking and single- vs. multi-token hashtag classification. The latter leads
to an error reduction of 24.6% over the current state-of-the-art. Finally, we
demonstrate the utility of our method by using hashtag segmentation in the
downstream task of sentiment analysis. Feeding the automatically segmented
hashtags to a state-of-the-art sentiment analysis method on the SemEval 2017
benchmark dataset results in a 2.6% increase in the official metric for the
task.

##  2 Background and Preliminaries

Current approaches for hashtag segmentation can be broadly divided into three
categories: (a) gazeteer and rule based Maynard and Greenwood (2014); Declerck
and Lendvai (2015); Billal et al. (2016), (b) word boundary detection Çelebi
and Özgür (2017, 2016), and (c) ranking with language model and other features
Wang et al. (2011); Bansal et al. (2015); Berardi et al. (2011); Reuter et al.
(2016); Simeon et al. (2016). Hashtag segmentation approaches draw upon work
on compound splitting for languages such as German or Finnish Koehn and Knight
(2003) and word segmentation Peng and Schuurmans (2001) for languages with no
spaces between words such as Chinese Sproat and Shih (1990); Xue and Shen
(2003). Similar to our work, Bansal et al. Bansal et al. (2015) extract an
initial set of candidate segmentations using a sliding window, then rerank
them using a linear regression model trained on lexical, bigram and other
corpus-based features. The current state-of-the-art approach Çelebi and Özgür
(2017, 2016) uses maximum entropy and CRF models with a combination of
language model and hand-crafted features to predict if each character in the
hashtag is the beginning of a new word.

Generating Candidate Segmentations. Microsoft Word Breaker Wang et al. (2011)
is, among the existing methods, a strong baseline for hashtag segmentation, as
reported in Çelebi and Özgür (2017) and Bansal et al. (2015). It employs a
beam search algorithm to extract $k$ best segmentations as ranked by the
n-gram language model probability:

$\displaystyle Score^{LM}(s)$ $\displaystyle=\sum_{i=1}^{n}\log
P(w_{i}|w_{i-N+1}\dots w_{i-1})$

where $[w_{1},w_{2}\dots w_{n}]$ is the word sequence of segmentation $s$ and
$N$ is the window size. More sophisticated ranking strategies, such as
Binomial and word length distribution based ranking, did not lead to a further
improvement in performance Wang et al. (2011). The original Word Breaker was
designed for segmenting URLs using language models trained on web data. In
this paper, we reimplemented222To the best of our knowledge, Microsoft
discontinued its Word Breaker and Web Ngram API services in early 2018. and
tailored this approach to segmenting hashtags by using a language model
specifically trained on Twitter data (implementation details in §3.6). The
performance of this method itself is competitive with state-of-the-art methods
(evaluation results in §5.3). Our proposed pairwise ranking method will
effectively take the top $k$ segmentations generated by this baseline as
candidates for reranking.

However, in prior work, the ranking scores of each segmentation were
calculated independently, ignoring the relative order among the top $k$
candidate segmentations. To address this limitation, we utilize a pairwise
ranking strategy for the first time for this task and propose neural
architectures to model this.

##  3 Multi-task Pairwise Neural Ranking

We propose a multi-task pairwise neural ranking approach to better incorporate
and distinguish the relative order between the candidate segmentations of a
given hashtag. Our model adapts to address single- and multi-token hashtags
differently via a multi-task learning strategy without requiring additional
annotations. In this section, we describe the task setup and three variants of
pairwise neural ranking models (Figure 1).

hashtag ($h$) | #songsonghaddafisitunes
---|---
segmentation ($s^{*}$) |  songs on ghaddafi s itunes
(i.e. songs on Ghaddafi’s iTunes)
candidate segmentations ($s\in S$)
|  songs on ghaddafis itunes
|  songs on ghaddafisi tunes
|  songs on ghaddaf is itunes
|  song song haddafis i tunes
|  songsong haddafisitunes
| (and $\dots$)
Table 2: Example hashtag along with its gold and possible candidate
segmentations.

###  3.1 Segmentation as Pairwise Ranking

The goal of hashtag segmentation is to divide a given hashtag $h$ into a
sequence of meaningful words $s^{*}=[w_{1},w_{2},\dots,w_{n}]$. For a hashtag
of $r$ characters, there are a total of $2^{r-1}$ possible segmentations but
only one, or occasionally two, of them ($s^{*}$) are considered correct (Table
2).

We transform this task into a pairwise ranking problem: given $k$ candidate
segmentations {$s_{1},s_{2},\ldots,s_{k}$}, we rank them by comparing each
with the rest in a pairwise manner. More specifically, we train a model to
predict a real number $g(s_{a},s_{b})$ for any two candidate segmentations
$s_{a}$ and $s_{b}$ of hashtag $h$, which indicates $s_{a}$ is a better
segmentation than $s_{b}$ if positive, and vice versa. To quantify the quality
of a segmentation in training, we define a gold scoring function $g^{*}$ based
on the similarities with the ground-truth segmentation $s^{*}$:

$g^{*}(s_{a},s_{b})=sim(s_{a},s^{*})-sim(s_{b},s^{*}).$

We use the Levenshtein distance (minimum number of single-character edits) in
this paper, although it is possible to use other similarity measurements as
alternatives. We use the top $k$ segmentations generated by Microsoft Word
Breaker (§2) as initial candidates.

(a) Pairwise Ranking Model (MSE §3.2)

(b) Margin Ranking Loss w/ shared parameters (MR §3.3)

(c) Adaptive Multi-task Learning for Pairwise ranking (MSE+Multitask §3.4)

Figure 1: Pairwise neural ranking models for hashtag segmentation. Given two
candidate segmentations $s_{a}$ and $s_{b}$ of hashtag $h$, the goal is to
predict the segmentation’s goodness relative score ($g$) or absolute
($g^{\prime}$) score.

###  3.2 Pairwise Neural Ranking Model

For an input candidate segmentation pair $\langle s_{a},s_{b}\rangle$, we
concatenate their feature vectors $\mathbf{s}_{a}$ and $\mathbf{s}_{b}$, and
feed them into a feedforward network which emits a comparison score
$g(s_{a},s_{b})$. The feature vector $\mathbf{s}_{a}$ or $\mathbf{s}_{b}$
consists of language model probabilities using Good-Turing Good (1953) and
modified Kneser-Ney smoothing Kneser and Ney (1995); Chen and Goodman (1999),
lexical and linguistic features (more details in §3.5). For training, we use
all the possible pairs $\langle s_{a},s_{b}\rangle$ of the $k$ candidates as
the input and their gold scores $g^{*}(s_{a},s_{b})$ as the target. The
training objective is to minimize the Mean Squared Error (MSE):

$L_{MSE}=\frac{1}{m}\sum_{i=1}^{m}(g^{*(i)}(s_{a},s_{b})-\hat{g}^{(i)}(s_{a},s_{b}))^{2}$
(1)

where $m$ is the number of training examples.

To aggregate the pairwise comparisons, we follow a greedy algorithm proposed
by Cohen et al. Cohen et al. (1998) and used for preference ranking Parakhin
and Haluptzok (2009). For each segmentation $s$ in the candidate set
$S=\\{s_{1},s_{2},\dots,s_{k}\\}$, we calculate a single score
$Score^{PNR}(s)=\sum_{s\neq s_{j}\in S}g(s,s_{j})$, and find the segmentation
$s_{max}$ corresponding to the highest score. We repeat the same procedure
after removing $s_{max}$ from $S$, and continue until $S$ reduces to an empty
set. Figure 1(a) shows the architecture of this model.

###  3.3 Margin Ranking (MR) Loss

As an alternative to the pairwise ranker (§3.2), we propose a pairwise model
which learns from candidate pairs $\langle s_{a},s_{b}\rangle$ but ranks each
individual candidate directly rather than relatively. We define a new scoring
function $g^{\prime}$ which assigns a higher score to the better candidate,
i.e., $g^{\prime}(s_{a})>g^{\prime}(s_{b})$, if $s_{a}$ is a better candidate
than $s_{b}$ and vice-versa. Instead of concatenating the features vectors
$\mathbf{s}_{a}$ and $\mathbf{s}_{b}$, we feed them separately into two
identical feedforward networks with shared parameters. During testing, we use
only one of the networks to rank the candidates based on the $g^{\prime}$
scores. For training, we add a ranking layer on top of the networks to measure
the violations in the ranking order and minimize the Margin Ranking Loss (MR):

$\displaystyle L_{MR}$
$\displaystyle=\frac{1}{m}\sum_{i=1}^{m}\max(0,1-l_{ab}^{(i)}p_{ab}^{(i)})$
(2) $\displaystyle p_{ab}^{(i)}$
$\displaystyle=(\hat{g}^{\prime(i)}(s_{a})-\hat{g}^{\prime(i)}(s_{b}))$
$\displaystyle l_{ab}$ $\displaystyle=\left\\{\begin{array}[]{ll}1&\quad
g^{*}(s_{a},s_{b})>0\\\ -1&\quad g^{*}(s_{a},s_{b})<0\\\
0&\quad\text{otherwise}\\\ \end{array}\right.$

where $m$ is the number of training samples. The architecture of this model is
presented in Figure 1(b).

###  3.4 Adaptive Multi-task Learning

Both models in §3.2 and §3.3 treat all the hashtags uniformly. However,
different features address different types of hashtags. By design, the
linguistic features capture named entities and multi-word hashtags that
exhibit word shape patterns, such as camel case. The ngram probabilities with
Good-Turing smoothing gravitate towards multi-word segmentations with known
words, as its estimate for unseen ngrams depends on the fraction of ngrams
seen once which can be very low Heafield (2013). The modified Kneser-Ney
smoothing is more likely to favor segmentations that contain rare words, and
single-word segmentations in particular. Please refer to §5.3 for a more
detailed quantitative and qualitative analysis.

To leverage this intuition, we introduce a binary classification task to help
the model differentiate single-word from multi-word hashtags. The binary
classifier takes hashtag features $\mathbf{h}$ as the input and outputs
$w_{h}$, which represents the probability of $h$ being a multi-word hashtag.
$w_{h}$ is used as an adaptive gating value in our multi-task learning setup.
The gold labels for this task are obtained at no extra cost by simply
verifying whether the ground-truth segmentation has multiple words. We train
the pairwise segmentation ranker and the binary single- vs. multi-token
hashtag classifier jointly, by minimizing $L_{MSE}$ for the pairwise ranker
and the Binary Cross Entropy Error ($L_{BCE}$) for the classifier:

$\displaystyle L_{multitask}$
$\displaystyle=\lambda_{1}L_{MSE}+\lambda_{2}L_{BCE}$ (3) $\displaystyle
L_{BCE}$
$\displaystyle=-\frac{1}{m}\sum_{i=1}^{m}\big{[}l^{(i)}*log(w_{h}^{(i)})+$
$\displaystyle(1-l^{(i)})*log(1-w_{h}^{(i)})\big{]}$

where $w_{h}$ is the adaptive gating value, $l\in\\{0,1\\}$ indicates if $h$
is actually a multi-word hashtag and $m$ is the number of training examples.
$\lambda_{1}$ and $\lambda_{2}$ are the weights for each loss. For our
experiments, we apply equal weights.

More specifically, we divide the segmentation feature vector $\mathbf{s}_{a}$
into two subsets: (a) $\mathbf{s}_{a}^{KN}$ with modified Kneser-Ney smoothing
features, and (b) $\mathbf{s}_{a}^{GL}$ with Good-Turing smoothing and
linguistic features. For an input candidate segmentation pair $\langle
s_{a},s_{b}\rangle$, we construct two pairwise vectors
$\mathbf{s}_{ab}^{KN}=[\mathbf{s}_{a}^{KN};\mathbf{s}_{b}^{KN}]$ and
$\mathbf{s}_{ab}^{GL}=[\mathbf{s}_{a}^{GL};\mathbf{s}_{b}^{GL}]$ by
concatenation, then combine them based on the adaptive gating value $w_{h}$
before feeding them into the feedforward network $G$ for pairwise ranking:

$\hat{g}(s_{a},s_{b})=G\left(w_{h}\mathbf{s}_{ab}^{GL}+(1-w_{h})\mathbf{s}_{ab}^{KN}\right)$
(4)

We use summation with padding, as we find this simple ensemble method achieves
similar performance in our experiments as the more complex multi-column
networks Ciresan et al. (2012). Figure 1(c) shows the architecture of this
model. An analogue multi-task formulation can also be used for the Margin
Ranking loss as:

$L_{multitask}=\lambda_{1}L_{MR}+\lambda_{2}L_{BCE}.$ (5)

###  3.5 Features

We use a combination of corpus-based and linguistic features to rank the
segmentations. For a candidate segmentation $s$, its feature vector
$\mathbf{s}$ includes the number of words in the candidate, the length of each
word, the proportion of words in an English
dictionary333https://pypi.org/project/pyenchant or Urban
Dictionary444https://www.urbandictionary.com Nguyen et al. (2018), ngram
counts from Google Web 1TB corpus Brants and Franz (2006), and ngram
probabilities from trigram language models trained on the Gigaword corpus
Graff and Cieri (2003) and 1.1 billion English tweets from 2010, respectively.
We train two language models on each corpus: one with Good-Turing smoothing
using SRILM Stolcke (2002) and the other with modified Kneser-Ney smoothing
using KenLM Heafield (2011). We also add boolean features, such as if the
candidate is a named-entity present in the list of Wikipedia titles, and if
the candidate segmentation $s$ and its corresponding hashtag $h$ satisfy
certain word-shapes (more details in appendix A.1).

Similarly, for hashtag $h$, we extract the feature vector $\mathbf{h}$
consisting of hashtag length, ngram count of the hashtag in Google 1TB corpus
Brants and Franz (2006), and boolean features indicating if the hashtag is in
an English dictionary or Urban Dictionary, is a named-entity, is in camel
case, ends with a number, and has all the letters as consonants. We also
include features of the best-ranked candidate by the Word Breaker model.

###  3.6 Implementation Details

We use the PyTorch framework to implement our multi-task pairwise ranking
model. The pairwise ranker consists of an input layer, three hidden layers
with eight nodes in each layer and hyperbolic tangent ($tanh$) activation, and
a single linear output node. The auxiliary classifier consists of an input
layer, one hidden layer with eight nodes and one output node with sigmoid
activation. We use the Adam algorithm Kingma and Ba (2014) for optimization
and apply a dropout of 0.5 to prevent overfitting. We set the learning rate to
0.01 and 0.05 for the pairwise ranker and auxiliary classifier respectively.
For each experiment, we report results obtained after 100 epochs.

For the baseline model used to extract the $k$ initial candidates, we
reimplementated the Word Breaker Wang et al. (2011) as described in §2 and
adapted it to use a language model trained on 1.1 billion tweets with Good-
Turing smoothing using SRILM Stolcke (2002) to give a better performance in
segmenting hashtags (§5.3). For all our experiments, we set $k=10$.

##  4 Hashtag Segmentation Data

We use two datasets for experiments (Table 3): (a) STANsmall, created by
Bansal et al. Bansal et al. (2015), which consists of 1,108 unique English
hashtags from 1,268 randomly selected tweets in the Stanford Sentiment
Analysis Dataset Go and Huang (2009) along with their crowdsourced
segmentations and our additional corrections; and (b) STANlarge, our new
expert curated dataset, which includes all 12,594 unique English hashtags and
their associated tweets from the same Stanford dataset.

#### Dataset Analysis.

STANsmall is the most commonly used dataset in previous work. However, after
reexamination, we found annotation errors in 6.8%555More specifically, 4.8%
hashtags is missing one of the two acceptable segmentations and another 2.0%
is incorrect segmentation. of the hashtags in this dataset, which is
significant given that the error rate of the state-of-the-art models is only
around 10%. Most of the errors were related to named entities. For example,
#lionhead, which refers to the “Lionhead” video game company, was labeled as
“lion head”.

| Data | num. of Hashtags | avg. | avg.
---|---|---|---|---
| (multi-token%) | #char | #word
| Train | 2518 (51.9%) | 8.5 | 1.8
STANlarge | Dev | 629 (52.3%) | 8.4 | 1.7
| Test | 9447 (53.0%) | 8.6 | 1.8
STANsmall | Test | 1108 (60.5%) | 9.0 | 1.9
Table 3: Statistics of the STANsmall and STANlarge datasets – number of unique
hashtags, percentage of multi-token hashtags, average length of hashtags in
characters and words.

#### Our Dataset.

We therefore constructed the STANlarge dataset of 12,594 hashtags with
additional quality control for human annotations. We displayed a tweet with
one highlighted hashtag on the Figure-Eight666https://figure-eight.com
(previously known as CrowdFlower) crowdsourcing platform and asked two workers
to list all the possible segmentations. For quality control on the platform,
we displayed a test hashtag in every page along with the other hashtags. If
any annotator missed more than 20% of the test hashtags, then they were not
allowed to continue work on the task. For 93.1% of the hashtags, out of which
46.6% were single-token, the workers agreed on the same segmentation. We
further asked three in-house annotators (not authors) to cross-check the
crowdsourced annotations using a two-step procedure: first, verify if the
hashtag is a named entity based on the context of the tweet; then search on
Google to find the correct segmentation(s). We also asked the same annotators
to fix the errors in STANsmall. The human upperbound of the task is estimated
at $\sim$98% accuracy, where we consider the crowdsourced segmentations (two
workers merged) as correct if at least one of them matches with our expert’s
segmentations.

| All Hashtags |  | Multi-token |  | Single-token
---|---|---|---|---|---
| A@1 | F1@1 | A@2 | MRR |  | A@1 | F1@1 | A@2 | MRR |  | A@1 | A@2 | MRR
Original hashtag | 51.0 | 51.0 | – | – |  | 19.1 | 19.1 | – | – |  | 100.0 | – | –
Rule-based Billal et al. (2016) | 58.1 | 63.5 | – | – |  | 57.6 | 66.5 | – | – |  | 58.8 | – | –
GATE Hashtag Tokenizer (M&G, 2014) | 73.2 | 77.2 | – | – |  | 71.4 | 78.0 | – | – |  | 76.0 | – | –
Viterbi Berardi et al. (2011) | 73.4 | 78.5 | – | – |  | 74.5 | 83.1 | – | – |  | 71.6 | – | –
MaxEnt Çelebi and Özgür (2017) | 92.4 | 93.4 | – | – |  | 91.9 | 93.6 | – | – |  | 93.1 | – | –
Word Breaker w/ Twitter LM | 90.8 | 91.7 | 97.4 | 94.5 |  | 88.5 | 90.0 | 97.8 | 93.7 |  | 94.3 | 96.8 | 95.7
Pairwise linear ranker | 88.1 | 89.9 | 97.2 | 93.1 |  | 83.8 | 86.8 | 97.3 | 91.3 |  | 94.7 | 97.0 | 95.9
Pairwise neural ranker (MR) | 92.3 | 93.5 | 98.2 | 95.4 |  | 90.9 | 92.8 | 99.0 | 95.2 |  | 94.5 | 96.9 | 95.8
Pairwise neural ranker (MSE) | 92.5 | 93.7 | 98.2 | 95.5 |  | 91.2 | 93.1 | 99.0 | 95.4 |  | 94.5 | 97.0 | 95.8
Pairwise neural ranker (MR+multitask) | 93.0 | 94.3 | 97.8 | 95.7 |  | 91.5 | 93.7 | 98.7 | 95.4 |  | 95.2 | 96.6 | 96.0
Pairwise neural ranker (MSE+multitask) | 94.5 | 95.2 | 98.4 | 96.6 |  | 93.9 | 95.1 | 99.4 | 96.8 |  | 95.4 | 96.8 | 96.2
Human Upperbound | 98.0 | 98.3 | – | – |  | 97.8 | 98.2 | – | – |  | 98.4 | – | –
Table 4: Evaluation results on the corrected version of STANsmall. For reference, on the original version of STANsmall, the Microsoft Word Breaker API reported an 84.6% F1 score and an 83.6% accuracy for the top one output Çelebi and Özgür (2017), while our best model (MSE+multitask) reported 89.8% F1 and 91.0% accuracy. | All Hashtags |  | Multi-token |  | Single-token
---|---|---|---|---|---
| A@1 | F1@1 | A@2 | MRR |  | A@1 | F1@1 | A@2 | MRR |  | A@1 | A@2 | MRR
Original hashtag | 55.5 | 55.5 | – | – |  | 16.2 | 16.2 | – | – |  | 100.0 | – | –
Rule-based Billal et al. (2016) | 56.1 | 61.5 | – | – |  | 56.0 | 65.8 | – | – |  | 56.3 | – | –
Viterbi Berardi et al. (2011) | 68.4 | 73.8 | – | – |  | 71.2 | 81.5 | – | – |  | 65.0 | – | –
GATE Hashtag Tokenizer (M&G, 2014) | 72.4 | 76.1 | – | – |  | 70.0 | 76.8 | – | – |  | 75.3 | – | –
MaxEnt Çelebi and Özgür (2017) | 91.2 | 92.3 | – | – |  | 90.2 | 92.4 | – | – |  | 92.3 | – | –
Word Breaker w/ Twitter LM | 90.1 | 91.0 | 96.6 | 93.9 |  | 88.5 | 90.0 | 97.0 | 93.4 |  | 91.9 | 96.2 | 94.4
Pairwise linear ranker | 89.2 | 91.1 | 96.3 | 93.3 |  | 84.2 | 87.8 | 95.6 | 91.0 |  | 94.8 | 97.0 | 95.9
Pairwise neural ranker (MR) | 91.3 | 92.6 | 97.2 | 94.6 |  | 89.9 | 92.4 | 97.5 | 94.3 |  | 92.8 | 96.8 | 94.9
Pairwise neural ranker (MSE) | 91.3 | 92.6 | 97.0 | 94.5 |  | 91.0 | 93.6 | 97.7 | 94.9 |  | 91.5 | 96.2 | 94.1
Pairwise neural ranker (MR+multitask) | 91.4 | 92.7 | 97.2 | 94.6 |  | 90.0 | 92.6 | 97.7 | 94.4 |  | 92.9 | 96.6 | 94.9
Pairwise neural ranker (MSE+multitask) | 92.4 | 93.6 | 97.3 | 95.2 |  | 91.9 | 94.1 | 98.0 | 95.4 |  | 93.0 | 96.5 | 94.9
Human Upperbound | 98.6 | 98.8 | – | – |  | 98.0 | 98.4 | – | – |  | 99.2 | – | –
Table 5: Evaluation results on our STANlarge test dataset. For single-token
hashtags, the token-level F1@1 is equivalent to segmentation-level A@1. For
multi-token cases, A@1 and F1@1 for the original hashtag baseline are non-zero
because 11.4% of the hashtags have more than one acceptable segmentations. Our
best model (MSE+multitask) shows a statistically significant improvement
($p<0.05$) over the state-of-the-art approach Çelebi and Özgür (2017) based on
the paired bootstrap test Berg-Kirkpatrick et al. (2012).

##  5 Experiments

In this section, we present experimental results that compare our proposed
method with the other state-of-the-art approaches on hashtag segmentation
datasets. The next section will show experiments of applying hashtag
segmentation to the popular task of sentiment analysis.

###  5.1 Existing Methods

We compare our pairwise neural ranker with the following baseline and state-
of-the-art approaches:

  1. (a)

The original hashtag as a single token;

  2. (b)

A rule-based segmenter, which employs a set of word-shape rules with an
English dictionary Billal et al. (2016);

  3. (c)

A Viterbi model which uses word frequencies from a book corpus777Project
Gutenberg http://norvig.com/big.txt Berardi et al. (2011);

  4. (d)

The specially developed GATE Hashtag Tokenizer from the open source
toolkit,888https://gate.ac.uk/ which combines dictionaries and gazetteers in a
Viterbi-like algorithm Maynard and Greenwood (2014);

  5. (e)

A maximum entropy classifier (MaxEnt) trained on the STANlarge training
dataset. It predicts whether a space should be inserted at each position in
the hashtag and is the current state-of-the-art Çelebi and Özgür (2017);

  6. (f)

Our reimplementation of the Word Breaker algorithm which uses beam search and
a Twitter ngram language model Wang et al. (2011);

  7. (g)

A pairwise linear ranker which we implemented for comparison purposes with the
same features as our neural model, but using perceptron as the underlying
classifier Hopkins and May (2011) and minimizing the hinge loss between
$g^{*}$ and a scoring function similar to $g^{\prime}$. It is trained on the
STANlarge dataset.

###  5.2 Evaluation Metrics

We evaluate the performance by the top $k$ ($k=1,2$) accuracy (A@1, A@2),
average token-level F1 score (F1@1), and mean reciprocal rank (MRR). In
particular, the accuracy and MRR are calculated at the segmentation-level,
which means that an output segmentation is considered correct if and only if
it fully matches the human segmentation. Average token-level F1 score accounts
for partially correct segmentation in the multi-token hashtag cases.

| Single |  | Multi |  | All
---|---|---|---|---|---
| A | MRR |  | A | MRR |  | A | MRR
Kneser-Ney | 95.4 | 95.7 |  | 56.0 | 75.3 |  | 74.9 | 85.1
Good-Turing (GT) | 91.4 | 93.5 |  | 85.9 | 91.8 |  | 88.6 | 92.6
Linguistic (Ling) | 89.4 | 91.7 |  | 71.6 | 82.6 |  | 80.1 | 87.0
GT + Ling | 92.4 | 93.9 |  | 86.2 | 92.3 |  | 88.9 | 92.7
All Features | 91.1 | 93.1 |  | 89.0 | 93.7 |  | 90.0 | 93.4
Table 6: Evaluation of automatic hashtag segmentation (MSE) with different
features on the STANlarge dev set. A denotes accuracy@1. While Kneser-Ney
features perform well on single-token hashtags, GT+Ling features perform
better on multi-token hashtags.

###  5.3 Results

Tables 4 and 5 show the results on the STANsmall and STANlarge datasets,
respectively. All of our pairwise neural rankers are trained on the 2,518
manually segmented hashtags in the training set of STANlarge and perform
favorably against other state-of-the-art approaches. Our best model
(MSE+multitask) that utilizes different features adaptively via a multi-task
learning procedure is shown to perform better than simply combining all the
features together (MR and MSE). We highlight the 24.6% error reduction on
STANsmall and 16.5% on STANlarge of our approach over the previous SOTA Çelebi
and Özgür (2017) on the Multi-token hashtags, and the importance of having a
separate evaluation of multi-word cases as it is trivial to obtain 100%
accuracy for Single-token hashtags. While our hashtag segmentation model is
achieving a very high accuracy@2, to be practically useful, it remains a
challenge to get the top one predication exactly correct. Some hashtags are
very difficult to interpret, e.g., #BTVSMB refers to the Social Media
Breakfast (SMB) in Burlington, Vermont (BTV).

The improved Word Breaker with our addition of a Twitter-specific language
model is a very strong baseline, which echos the findings of the original Word
Breaker paper Wang et al. (2011) that having a large in-domain language model
is extremely helpful for word segmentation tasks. It is worth noting that the
other state-of-the-art system Çelebi and Özgür (2017) also utilized a 4-gram
language model trained on 476 million tweets from 2009.

Kneser-Ney |  Good-Turing |  Linguistic |  | count |  | Example Hashtags |
---|---|---|---|---|---|---|---
$\circ$ | $\circ$ | $\circ$ |  | 31 |  |  #omnomnom #BTVSMB |
$\bullet$ | $\circ$ | $\circ$ |  | 13 |  |  #commbank #mamapedia |
$\circ$ | $\bullet$ | $\circ$ |  | 38 |  |  #wewantmcfly #winebarsf |
$\circ$ | $\circ$ | $\bullet$ |  | 24 |  | #cfp09 #TechLunchSouth |
$\bullet$ | $\bullet$ | $\circ$ |  | 44 |  |  #twittographers #bringback |
$\bullet$ | $\circ$ | $\bullet$ |  | 16 |  |  #iccw #ecom09 |
$\circ$ | $\bullet$ | $\bullet$ |  | 53 |  |  #LetsGoPens #epicwin |
$\bullet$ | $\bullet$ | $\bullet$ |  | 420 |  |  #prototype #newyork |
Table 7: Error ($\circ$) and correct ($\bullet$) segmentation analysis of
three pairwise ranking models (MSE) trained with different feature sets Each
row corresponds to one area in the Venn diagram; for example,
$\circ$$\circ$$\circ$ is the set of hashtags that all three models failed in
the STANlarge dev data and $\bullet$$\circ$$\circ$ is the set of hashtags that
only the model with Kneser-Ney language model features (but not the other two
models) segmented correctly.

###  5.4 Analysis and Discussion

#### Feature Analysis.

To empirically illustrate the effectiveness of different features on different
types of hashtags, we show the results for models using individual feature
sets in pairwise ranking models (MSE) in Table 6. Language models with
modified Kneser-Ney smoothing perform best on single-token hashtags, while
Good-Turing and Linguistic features work best on multi-token hashtags,
confirming our intuition about their usefulness in a multi-task learning
approach. Table 7 shows a qualitative analysis with the first column
($\circ$$\circ$$\circ$) indicating which features lead to correct or wrong
segmentations, their count in our data and illustrative examples with human
segmentation.

#### Length of Hashtags.

As expected, longer hashtags with more than three tokens pose greater
challenges and the segmentation-level accuracy of our best model
(MSE+multitask) drops to 82.1%. For many error cases, our model predicts a
close-to-correct segmentation, e.g., #youknowyouupttooearly,
#iseelondoniseefrance, which is also reflected by the higher token-level F1
scores across hashtags with different lengths (Figure 2).

Type | num. of Hashtags
---|---
single | 4426 (47.1%)
2 tokens | 3436 (36.2%)
3 tokens | 1085 (11.2%)
4 tokens | 279 (2.9%)
5+ tokens | 221 (2.6%)

Figure 2: Token-level F1 scores (MSE+multitask) on hashtags of different
lengths in the STANlarge test set.  Figure 3: Token-level F1 scores of our
pairwise ranker (MSE+multitask) and Word Breaker on the STANlarge test set,
using language models trained with varying amounts of data.

#### Size of the Language Model.

Since our approach heavily relies on building a Twitter language model, we
experimented with its sizes and show the results in Figure 3. Our approach can
perform well even with access to a smaller amount of tweets. The drop in F1
score for our pairwise neural ranker is only 1.4% and 3.9% when using the
language models trained on 10% and 1% of the total 1.1 billion tweets,
respectively.

#### Time Sensitivity.

Language use in Twitter changes with time Eisenstein (2013). Our pairwise
ranker uses language models trained on the tweets from the year 2010. We
tested our approach on a set of 500 random English hashtags posted in tweets
from the year 2019 and show the results in Table 8. With a segmentation-level
accuracy of 94.6% and average token-level F1 score of 95.6%, our approach
performs favorably on 2019 hashtags.

|  A@1 |  F1@1 |  MRR
---|---|---|---
Word Breaker w/ Twitter LM |  92.1 |  93.9 |  94.7
Pairwise neural ranker (MSE+multitask) |  94.6 |  95.6 |  96.7
Table 8: Evaluation results on 500 random hashtags from the year 2019.

##  6 Extrinsic Evaluation: Twitter Sentiment Analysis

We attempt to demonstrate the effectiveness of our hashtag segmentation system
by studying its impact on the task of sentiment analysis in Twitter Pang et
al. (2002); Nakov et al. (2016); Rosenthal et al. (2017). We use our best
model (MSE+multitask), under the name HashtagMaster, in the following
experiments.

###  6.1 Experimental Setup

We compare the performance of the BiLSTM+Lex Teng et al. (2016) sentiment
analysis model under three configurations: (a) tweets with hashtags removed,
(b) tweets with hashtags as single tokens excluding the # symbol, and (c)
tweets with hashtags as segmented by our system, HashtagMaster. BiLSTM+Lex is
a state-of-the-art open source system for predicting tweet-level sentiment Tay
et al. (2018). It learns a context-sensitive sentiment intensity score by
leveraging a Twitter-based sentiment lexicon Tang et al. (2014). We use the
same settings as described by Teng et al. Teng et al. (2016) to train the
model.

We use the dataset from the Sentiment Analysis in Twitter shared task (subtask
A) at SemEval 2017 Rosenthal et al. (2017). 999We did not use the Stanford
Sentiment Analysis Dataset Go and Huang (2009), which was used to construct
the STANsmall and STANlarge hashtag datasets, because of its noisy sentiment
labels obtained using distant supervision. Given a tweet, the goal is to
predict whether it expresses POSITIVE, NEGATIVE or NEUTRAL sentiment. The
training and development sets consist of 49,669 tweets and we use 40,000 for
training and the rest for development. There are a total of 12,284 tweets
containing 12,128 hashtags in the SemEval 2017 test set, and our hashtag
segmenter ended up splitting 6,975 of those hashtags present in 3,384 tweets.

###  6.2 Results and Analysis

| AvgR | F${}_{1}^{PN}$ | Acc
---|---|---|---
Original tweets | 61.7 | 60.0 | 58.7
$-$ No Hashtags | 60.2 | 58.8 | 54.2
$+$ Single-word | 62.3 | 60.3 | 58.6
$+$ HashtagMaster | 64.3 | 62.4 | 58.6
Table 9: Sentiment analysis evaluation on the 3384 tweets from SemEval 2017
test set using the BiLSTM+Lex method Tang et al. (2014). Average recall (AvgR)
is the official metric of the SemEval task and is more reliable than accuracy
(Acc). F${}_{1}^{PN}$ is the average F1 of positive and negative classes.
Having the hashtags segmented by our system HashtagMaster (i.e.,
MSE+multitask) significantly improves the sentiment prediction than not
($p<0.05$ for AvgR and F${}_{1}^{PN}$ against the single-word setup).

In Table 9, we report the results based on the 3,384 tweets where
HashtagMaster predicted a split, as for the rest of tweets in the test set,
the hashtag segmenter would neither improve nor worsen the sentiment
prediction. Our hashtag segmenter successfully improved the sentiment analysis
performance by 2% on average recall and F${}_{1}^{PN}$ comparing to having
hashtags unsegmented. This improvement is seemingly small but decidedly
important for tweets where sentiment-related information is embedded in multi-
word hashtags and sentiment prediction would be incorrect based only on the
text (see Table 10 for examples). In fact, 2,605 out of the 3,384 tweets have
multi-word hashtags that contain words in the Twitter-based sentiment lexicon
Tang et al. (2014) and 125 tweets contain sentiment words only in the hashtags
but not in the rest of the tweet. On the entire test set of 12,284 tweets, the
increase in the average recall is 0.5%.

##  7 Other Related Work

Automatic hashtag segmentation can improve the performance of many
applications besides sentiment analysis, such as text classification Billal et
al. (2016), named entity linking Bansal et al. (2015) and modeling user
interests for recommendations Chen et al. (2016). It can also help in
collecting data of higher volume and quality by providing a more nuanced
interpretation of its content, as shown for emotion analysis Qadir and Riloff
(2014), sarcasm and irony detection Maynard and Greenwood (2014); Huang et al.
(2018). Better semantic analysis of hashtags can also potentially be applied
to hashtag annotation Wang et al. (2019), to improve distant supervision
labels in training classifiers for tasks such as sarcasm Bamman and Smith
(2015), sentiment Mohammad et al. (2013), emotions Abdul-Mageed and Ungar
(2017); and, more generally, as labels for pre-training representations of
words Weston et al. (2014), sentences Dhingra et al. (2016), and images
Mahajan et al. (2018).

|  Ofcourse #clownshoes #altright #IllinoisNazis |
---|---|---
|  #FinallyAtpeaceWith people calling me “Kim Fatty the Third” |
|  Leslie Odom Jr. sang that. #ThankYouObama |
|  After some 4 months of vegetarianism .. it’s all the same industry. #cutoutthecrap |
Table 10: Sentiment analysis examples where our HashtagMaster segmentation
tool helped. Red and blue words are negative and positive entries in the
Twitter sentiment lexicon Tang et al. (2014), respectively.

##  8 Conclusion

We proposed a new pairwise neural ranking model for hashtag segmention and
showed significant performance improvements over the state-of-the-art. We also
constructed a larger and more curated dataset for analyzing and benchmarking
hashtag segmentation methods. We demonstrated that hashtag segmentation helps
with downstream tasks such as sentiment analysis. Although we focused on
English hashtags, our pairwise ranking approach is language-independent and we
intend to extend our toolkit to languages other than English as future work.

## Acknowledgments

We thank Ohio Supercomputer Center Center (2012) for computing resources and
the NVIDIA for providing GPU hardware. We thank Alan Ritter, Quanze Chen, Wang
Ling, Pravar Mahajan, and Dushyanta Dhyani for valuable discussions. We also
thank the annotators: Sarah Flanagan, Kaushik Mani, and Aswathnarayan
Radhakrishnan. This material is based in part on research sponsored by the NSF
under grants IIS-1822754 and IIS-1755898, DARPA through the ARO under
agreement number W911NF-17-C-0095, through a Figure-Eight (CrowdFlower) AI for
Everyone Award and a Criteo Faculty Research Award to Wei Xu. The views and
conclusions contained in this publication are those of the authors and should
not be interpreted as representing official policies or endorsements of the
U.S. Government.

## References

  * Abdul-Mageed and Ungar (2017) Muhammad Abdul-Mageed and Lyle Ungar. 2017.  Emonet: Fine-grained emotion detection with gated recurrent neural networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , ACL, pages 718–728.
  * Bamman and Smith (2015) David Bamman and Noah A Smith. 2015.  Contextualized Sarcasm Detection on Twitter.  In _Ninth International AAAI Conference on Web and Social Media_ , ICWSM, pages 574–577.
  * Bansal et al. (2015) Piyush Bansal, Romil Bansal, and Vasudeva Varma. 2015.  Towards Deep Semantic Analysis of Hashtags.  In _Proceedings of the 37th European Conference on Information Retrieval_ , ECIR, pages 453–464.
  * Berardi et al. (2011) Giacomo Berardi, Andrea Esuli, Diego Marcheggiani, and Fabrizio Sebastiani. 2011\.  ISTI@TREC Microblog Track 2011: Exploring the Use of Hashtag Segmentation and Text Quality Ranking.  In _Text REtrieval Conference (TREC)_.
  * Berg-Kirkpatrick et al. (2012) Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012.  An Empirical Investigation of Statistical Significance in NLP.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , EMNLP-CoNLL, pages 995–1005.
  * Billal et al. (2016) Belainine Billal, Alexsandro Fonseca, and Fatiha Sadat. 2016.  Named Entity Recognition and Hashtag Decomposition to Improve the Classification of Tweets.  In _Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT)_ , COLING, pages 102–111.
  * Brants and Franz (2006) Thorsten Brants and Alex Franz. 2006.  Web 1T 5-gram Version 1.  _Linguistic Data Consortium (LDC)_.
  * Çelebi and Özgür (2016) Arda Çelebi and Arzucan Özgür. 2016.  Segmenting Hashtags using Automatically Created Training Data.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation_ , LREC, pages 2981–2985.
  * Çelebi and Özgür (2017) Arda Çelebi and Arzucan Özgür. 2017.  Segmenting Hashtags and Analyzing Their Grammatical Structure.  _Journal of Association For Information Science and Technology (JASIST)_ , 69(5):675–686.
  * Center (2012) Ohio Supercomputer Center. 2012.  Oakley supercomputer.  http://osc.edu/ark:/19495/hpc0cvqn.
  * Chen and Goodman (1999) Stanley F Chen and Joshua Goodman. 1999.  An empirical study of smoothing techniques for language modeling.  _Computer Speech & Language_, 13(4):359–394.
  * Chen et al. (2016) Tao Chen, Xiangnan He, and Min-Yen Kan. 2016.  Context-aware Image Tweet Modelling and Recommendation.  In _Proceedings of the 24th ACM International Conference on Multimedia_ , MM, pages 1018–1027.
  * Ciresan et al. (2012) Dan Ciresan, Ueli Meier, and Jürgen Schmidhuber. 2012.  Multi-column Deep Neural Networks for Image Classification.  In _Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition_ , CVPR, pages 3642–3649.
  * Cohen et al. (1998) William W Cohen, Robert E Schapire, and Yoram Singer. 1998.  Learning to Order Things.  In _Advances in Neural Information Processing Systems_ , NIPS, pages 451–457.
  * Declerck and Lendvai (2015) Thierry Declerck and Piroska Lendvai. 2015.  Processing and normalizing hashtags.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing_ , RANLP, pages 104–109.
  * Dhingra et al. (2016) Bhuwan Dhingra, Zhong Zhou, Dylan Fitzpatrick, Michael Muehl, and William Cohen. 2016.  Tweet2Vec: Character-Based Distributed Representations for Social Media.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_ , ACL, pages 269–274.
  * Eisenstein (2013) Jacob Eisenstein. 2013.  What to do about bad language on the Internet.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics_ , NAACL, pages 359–369.
  * Finin et al. (2010) Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010.  Annotating named entities in Twitter data with crowdsourcing.  In _Proceedings of the Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk_ , NAACL, pages 80–88.
  * Go and Huang (2009) Bhayani R. Go, A. and L. Huang. 2009.  Twitter Sentiment Classification using Distant Supervision.  _CS224N Project Report, Stanford_.
  * Good (1953) Irving J Good. 1953.  The population frequencies of species and the estimation of population parameters.  _Biometrika_ , 40(3-4):237–264.
  * Graff and Cieri (2003) David Graff and Christopher Cieri. 2003.  English Gigaword LDC2003T05.  _Linguistic Data Consortium (LDC)_.
  * Han and Baldwin (2011) Bo Han and Timothy Baldwin. 2011.  Lexical Normalisation of Short Text Messages: Makn Sens a# twitter.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics_ , ACL, pages 368–378.
  * Heafield (2011) Kenneth Heafield. 2011.  KenLM: Faster and Smaller Language Model Queries.  In _Proceedings of the Sixth Workshop on Statistical Machine Translation_ , WMT, pages 187–197.
  * Heafield (2013) Kenneth Heafield. 2013.  _Efficient Language Modeling Algorithms with Applications to Statistical Machine Translation_.  Ph.D. thesis, Carnegie Mellon University.
  * Hopkins and May (2011) Mark Hopkins and Jonathan May. 2011.  Tuning as ranking.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , EMNLP.
  * Huang et al. (2018) Hen-Hsen Huang, Chiao-Chen Chen, and Hsin-Hsi Chen. 2018.  Disambiguating false-alarm hashtag usages in tweets for irony detection.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , ACL, pages 771–777.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014.  Adam: A Method for Stochastic Optimization.  In _Proceedings of the 3rd International Conference for Learning Representations_ , ICLR.
  * Kneser and Ney (1995) Reinhard Kneser and Hermann Ney. 1995.  Improved backing-off for m-gram language modeling.  In _Proceedings of the 1995 International Conference on Acoustics, Speech, and Signal Processing_ , ICASSP, pages 181–184.
  * Koehn and Knight (2003) Philipp Koehn and Kevin Knight. 2003.  Empirical methods for compound splitting.  In _Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics_ , EACL, pages 187–194.
  * Mahajan et al. (2018) Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. 2018.  Exploring the Limits of Weakly Supervised Pretraining.  In _Tech Report_.
  * Maynard and Greenwood (2014) Diana Maynard and Mark A Greenwood. 2014.  Who cares about sarcastic tweets? Investigating the impact of sarcasm on sentiment analysis.  In _Proceedings of the 9th International Conference on Language Resources and Evaluation_ , LREC, pages 4238–4243.
  * Mohammad et al. (2013) Saif Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013.  NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets.  In _Proceedings of the Seventh International Workshop on Semantic Evaluation_ , SemEval, pages 321–327.
  * Nakov et al. (2016) Preslav Nakov, Sara Rosenthal, Svetlana Kiritchenko, Saif M. Mohammad, Zornitsa Kozareva, Alan Ritter, Veselin Stoyanov, and Xiaodan Zhu. 2016.  Developing a successful SemEval task in sentiment analysis of Twitter and other social media texts.  _Language Resources and Evaluation_ , 50(1):35–65.
  * Nguyen et al. (2018) Dong Nguyen, Barbara McGillivray, and Taha Yasseri. 2018.  Emo, love and god: making sense of urban dictionary, a crowd-sourced online dictionary.  _Royal Society Open Science_ , 5(5):172320.
  * Ozdikis et al. (2012) Ozer Ozdikis, Pinar Senkul, and Halit Oguztuzun. 2012.  Semantic Expansion of Hashtags for Enhanced Event Detection in Twitter.  In _Proceedings of the 1st international Workshop on Online Social Systems_.
  * Pang et al. (2002) Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002.  Thumbs up? Sentiment Classification using Machine Learning Techniques.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , EMNLP, pages 79–86.
  * Parakhin and Haluptzok (2009) M. Parakhin and P. Haluptzok. 2009.  Finding the Most Probable Rranking of Objects with Probabilistic Pairwise Preferences.  In _Proceedings of the 10th International Conference on Document Analysis and Recognition_ , ICDAR, pages 616–620.
  * Peng and Schuurmans (2001) Fuchun Peng and Dale Schuurmans. 2001.  A hierarchical em approach to word segmentation.  In _NLPRS_ , pages 475–480.
  * Qadir and Riloff (2014) Ashequl Qadir and Ellen Riloff. 2014.  Learning emotion indicators from tweets: Hashtags, hashtag patterns, and phrases.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_ , EMNLP, pages 1203–1209.
  * Reuter et al. (2016) Jack Reuter, Jhonata Pereira-Martins, and Jugal Kalita. 2016.  Segmenting twitter hashtags.  _International Journal on Natural Language Computing_ , 5:23–36.
  * Ritter et al. (2011) Alan Ritter, Sam Clark, Oren Etzioni, et al. 2011.  Named Entity Recognition in Tweets: An Experimental Study.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , EMNLP, pages 1524–1534.
  * Rosenthal et al. (2017) Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017.  SemEval-2017 task 4: Sentiment Analysis in Twitter.  In _Proceedings of the 11th International Workshop on Semantic Evaluation_ , SemEval, pages 502–518.
  * Sampson et al. (2016) Justin Sampson, Fred Morstatter, Liang Wu, and Huan Liu. 2016.  Leveraging the implicit structure within social media for emergent rumor detection.  In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_ , CIKM, pages 2377–2382.
  * Simeon et al. (2016) C. Simeon, H. J. Hamilton, and R. J. Hilderman. 2016.  Word segmentation algorithms with lexical resources for hashtag classification.  In _Proceedings of the 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA)_ , pages 743–751.
  * Sproat and Shih (1990) Richard Sproat and Chilin Shih. 1990.  A statistical method for finding word boundaries in chinese text.  _Computer Processing of Chinese and Oriental Languages_ , 4(4):336–351.
  * Stolcke (2002) Andreas Stolcke. 2002.  SRILM – An Extensible Language Modeling Toolkit.  In _Proceedings of the 7th International Conference on Spoken Language Processing_ , ICSLP, pages 901–904.
  * Tang et al. (2014) Duyu Tang, Furu Wei, Bing Qin, Ming Zhou, and Ting Liu. 2014.  Building Large-Scale Twitter-Specific Sentiment Lexicon : A Representation Learning Approach.  In _Proceedings of the 25th International Conference on Computational Linguistics_ , COLING, pages 172–182.
  * Tay et al. (2018) Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018.  Attentive gated lexicon reader with contrastive contextual co-attention for sentiment classification.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , EMNLP, pages 3443–3453.
  * Teng et al. (2016) Zhiyang Teng, Duy Tin Vo, and Yue Zhang. 2016.  Context-Sensitive Lexicon Features for Neural Sentiment Analysis.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , EMNLP, pages 1629–1638.
  * Wang et al. (2011) Kuansan Wang, Christopher Thrasher, and Bo-June Paul Hsu. 2011.  Web Scale NLP: A Case Study on URL Word Breaking.  In _Proceedings of the 20th International Conference on World Wide Web_ , WWW, pages 357–366.
  * Wang et al. (2019) Yue Wang, Jing Li, Irwin King, Michael R. Lyu, and Shuming Shi. 2019.  Microblog Hashtag Generation via Encoding Conversation Contexts.  In _Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Weston et al. (2014) Jason Weston, Sumit Chopra, and Keith Adams. 2014.  # tagspace: Semantic embeddings from hashtags.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_ , EMNLP, pages 1822–1827.
  * Xue and Shen (2003) Nianwen Xue and Libin Shen. 2003.  Chinese word segmentation as LMR tagging.  In _Proceedings of the second SIGHAN workshop on Chinese Language Processing_ , SIGHAN, pages 176–179.

##  Appendix A Appendix

###  A.1 Word-shape rules

Our model uses the following word shape rules as boolean features. If the
candidate segmentation $s$ and its corresponding hashtag $h$ satisfies a word
shape rule, then the boolean feature is set to True.

Rule | Hashtag $\rightarrow$ Segmentation
---|---
Camel Case | XxxXxx $\rightarrow$ Xxx$+$Xxx
Consonants | cccc $\rightarrow$ cccc
Digits as prefix | ddwwww $\rightarrow$ dd$+$wwww
Digits as suffix | wwwwdd $\rightarrow$ wwww$+$dd
Underscore | www$\textunderscore$www $\rightarrow$ www $+$ $\textunderscore$ $+$ www
Table 11: Word-shape rule features used to identify good segmentations. Here,
$X$ and $x$ represent capitalized and non-capitalized alphabetic characters
respectively, $c$ denotes consonant, $d$ denotes number and $w$ denotes any
alphabet or number.