# Voices of Workers: Why a Worker-Centered Approach to Crowd Work Is
Challenging

Caifan Du
School of Information
University of Texas at Austin
Austin, TX
<EMAIL_ADDRESS>
Matthew Lease
School of Information
University of Texas at Austin
Austin, Tx
<EMAIL_ADDRESS>

###### Abstract

How can we better understand the broad, diverse, shifting, and invisible crowd
workforce, so that we might better support it? We present findings from online
observations and analysis of publicly available postings from a community
forum of crowd workers. In particular, we observed recurring tensions between
crowd workers and journalists regarding media depictions of crowd work. We
found that crowd diversity makes any one-dimensional representation inadequate
in addressing the wide-ranging experiences of crowd work. We argue that the
scale, diversity, invisibility, and the crowds’ resistance to publicity make a
worker-centered approach to crowd work particularly challenging, necessitating
better understanding the diversity of workers and their lived experiences.

##  1 Introduction

Crowdsourcing platforms such as Amazon Mechanical Turk (AMT) provide online
marketplaces for task-based microwork [1, 2]. AMT in particular has been
widely used for research and business purposes [3], including tasks for
natural language processing, image annotation, online experiments, and user
studies (e.g., [4, 5, 6, 7, 8]). Existing studies have estimated over 100K
active workers on AMT and more than 2K working at any given time (e.g.,[9]).

Two primary classes of users engage in economic exchanges on the platform:
Requesters who distribute tasks and crowd workers who complete tasks for
payment. A decade ago, [10] called for more worker-centered research in this
domain. Since then, researchers have studied AMT worker demographics [11, 12],
motivation [13, 14, 15, 16, 17, 18], pay rates [19, 11, 20, 21, 22, 23, 24],
working experience [25, 18, 26, 27, 21], and their collective actions [28,
29]. While researchers have employed a wide range of methods to understand AMT
workers and their work, there has been a focus for eliciting workers’ own
perspective and voices, such as observing worker interactions in their online
community forums [18, 27], collecting worker responses and accounts through
surveys and interviews [25, 27, 30], and gathering requirements directly from
workers when building tools to support Turking, usually via "HITs" (i.e., unit
of work called "Human Intelligence Tasks" on AMT) [31, 30, 29].

We seek to build on this line of research by further enhancing the
understanding of why it can be so challenging to devise effective worker-
centered support mechanisms for crowd work. While there certainly have been
many valuable works in this vein, detailed reflections on the friction
involved are less common.

Through online observation in AMT worker community forums, we note tension
between workers and their media exposure. This tension prompted us to analyze
worker interactions and narratives posted in online discussion forums as
publicly available content. Our qualitative analysis sheds further light upon
the causes and mechanisms that render a worker-centered approach to crowd work
difficult. At its core, it is the challenge of effectively eliciting and
representing a plurality of diverse workers’ voices and perspectives [32, 33]
that make it difficult to speak for or design interventions that are
appreciated by these workers.

Because our findings are centered around crowd workers on AMT, we use the term
Turkers to refer to these workers, consistent with how these workers often
identify themselves, as we have observed in their online community forums.

##  2 Related Work

A variety of research has focused on learning about and improving the Turking
experience. Researchers who rely on the principles of ethnomethodology have
investigated how Turkers arrange their day-to-day work activities in
interaction with other geographically distributed workers, Requesters, and the
AMT platform as both an organizational entity and a task marketplace [18, 27].
[18] were among the very first to analyze the publicly available conversations
in Turkers’ online forums. They characterized Turkers as economic actors
operating in an online labor market, focusing on workplace issues such as
labor relations, wages, and work ethics. In [27]’s later study, they compared
US-based vs. India-based Turkers to better understand differences in working
experience across geographical boundaries. Similarly, [25] provides a detailed
description of Indian Turkers’ working lives, in contrast with that of US-
based workers based on surveys and interviews. Some specific aspects of
Turking in India include adapting to time zone differences and viewing Turking
work as an international professional experience in individual career
profiles.

Critical designers [31] have sought to support Turkers via system building
efforts, eliciting Turker voices for design ideation. [31] conducted an open-
ended survey as HIT on AMT, getting input from Turkers for compiling Workers’
Bill of Rights. They asked about daily work concerns and then developed a
Requester rating system, _Turkopticon_ , based on Turker responses. [34]
helped to introduce Turkers into HCI conference venues for their better
visibility. They also reflect on their experience of constructing and
maintaining the technological system as well as building relations with
Turkers. Related to our work, they note that the lack of unity in worker
voices in online forums makes it challenging to form collective voices and
actions, reflecting the difficulty of worker-centered design and advocacy.

[29] seeks to raise Turkers’ voices by organizing collective actions within
the Turker community. With the help of the researchers, Turkers collectively
authored action guidelines for academic Requesters and a letter to Amazon that
advocated fairness and openness toward Turkers [35]. Both efforts are
mobilized campaigns following the ideation by the Dynamo Turker community.
Other researchers [36, 30] have formulated technological interventions to help
workers earn a minimum hourly wage. Both research groups also surveyed workers
via HITs on AMT, e.g., regarding their expectations for fair pay. More recent
studies of Turking experience and designing for supporting Turking dominantly
rely on Turkers’ online comments and HIT surveys [37, 38, 39, 40].

Prior worker-centered studies range from observation of Turkers’ online
conversations to surveying and interviewing Turkers directly and using their
responses to design and develop supportive tools. In particular, active
critical designers seek to interrupt worker invisibility [31], helping them to
form voices, be heard, and be seen. As all these works have explored some
channels to get Turkers’ voices and experiences, it is not surprising that it
is often the invisibility of Turking that motivates these studies and affects
their interpretations. Similarly, [15] characterize invisible crowd work as
ghost work through five years of study. Our work builds upon such past
scholarship in seeking to provide explanations for the ways in which
invisibility operates to make a work-centered approach to crowd work
particularly challenging.

Regarding the common popular press narrative of exploited Turkers, [41]
presents a data-driven rebuttal that resonates with our own analysis.

##  3 Methods

Our work was prompted by initial observation in Turkers’ discussion forums. We
began to frequent several known online gathering spaces of crowd workers –
keeping up with the recent updates and reading through historical discussion
threads – to gain a broader understanding of Turkers’ online activities and
discourse. Those sites include Turker
Nation111https://www.reddit.com/r/TurkerNation/,
TurkerView222https://forum.turkerview.com/, and several
Subreddits333https://www.reddit.com/reddits/ dedicated to discussions about
Turking or crowd work of all sorts.

We began to notice a set of similar discussion threads in Subreddit
r/mturk444https://www.reddit.com/r/mturk/, an AMT forum (having more than 80K
members as of September 2021) in which Turkers discuss their daily Turking
experiences. Topics span issues such as how to make sense of posted HITs, how
to understand and manage rejections and get payments transferred to a bank
account, etc. The Subreddit thus serves as a gathering place for Turkers to
help each other navigate the dynamics of AMT work.

The set of discussion threads we focused on involve conversations between
self-identified journalists and Turkers. Threads were initiated by a
journalist, or occasionally a researcher, asking whether Turkers in the forum
would participate in an interview to discuss their Turking experiences, for a
news article or research on crowd work. The first reply in these threads
frequently conveyed some degree of resistance to being interviewed. We began
to read through and interpret these discussion threads as recurring events
observed in the field [42]. We also further searched the Subreddit using
journalist or interview as keywords, and our review of search results
identified more such threads.

As our observation and qualitative analysis went on, we further noticed that
community members routinely noted and discussed news media portrayals of them.
Many discussion threads stemmed from sharing of a new news story covering
Turkers, followed by discussion. As in the threads in which journalists sought
Turkers to interview, these discussions also contained Turker reactions to
news media publicizing them and their work. Searching through the historical
postings about news coverage of Turkers in the Subreddit, we identified
another set of 12 threads containing substantial discussions of how Turkers
themselves make sense of popular narratives about their images, conditions,
and experiences.

Overall, we identified 26 threads posted across the past six years, with the
latest posted in the last year. Most are substantial discussions; the shortest
has three comments while the longest has eighty-nine. These discussion threads
evidence similar events: Turkers’ reaction to the potential or ongoing
publicity about them. These 26 threads were initiated by 21 registered Reddit
user accounts; in 5 cases, an interview request was posted more than once as
separate threads, but with different conversations unfolding. Each discussion
for a thread involved an almost exclusively different set of community
members. Only five user accounts, as we identified, appeared in more than one
discussion thread.

We analyzed these discussion threads based on the principles of thematic
analysis, without any epistemological commitment preceding, coding each thread
inductively [43, 44]. We treated each thread as one conversation and paid
close attention to the role and stance of each participant involved. We
analyzed each comment in the thread and interpreted it in the context of the
conversation. We then memo’ed each comment with interpreted meaning as initial
codes. When one comment became sufficiently long to carry complex meanings, we
further decomposed it and memo’ed them with multiple codes accordingly. Next,
we sought converging themes across these codes and threads. As we organized
coded data under different themes, we reviewed the whole coded data set in
relation to identified themes to ensure mutual fit. A few codes were dropped
in this phase, carrying some special meaning but not connected to any themes,
or not supporting a theme-level finding given their sparse presence in the
data.

##  4 Voices of Workers

In this section, we discuss our themed findings from the identified data in
the field. For privacy, we anonymize the usernames used in Subreddit
discussions.

###  4.1 Conflicting voices

#### Telling stories is less attractive than making money.

As previously mentioned, in the first set of postings we identified, each
discussion thread started with a request from a journalist or researcher for
an interview with Turkers. The first reply from Turkers to each request seemed
to be not just subtly uncooperative, but also to convey a primary concern with
financial compensation when being requested to perform a task for free that
they might otherwise earn money from. Some examples of such first replies
include:

> _“Want work from a turker? Post a HIT on the platform.”_ [Turker1]
>
> _“If you really want turker’s opinions, you could set up a hit for it I’m
> sure.”_ [Turker2]
>
> _“How much does this HIT pay?_ [Turker3]
>
> _“Yeah, a lot of us posted wanting to know what you’re paying. We measure
> our lives by the minute.”_ [Turker4]
>
> _“What’s the pay/time commitment? I used to be a decent guy until I realized
> that an hour of time can mean the difference between being late on the
> electric bill.”_ [Turker5]

These initial responses showed no interest in sharing their working stories.
Instead, they seemed to view such requests as labor and equivalent to the paid
tasks they perform on AMT. Turker4 and Turker5 put their financial
consideration in tangible terms: for them, spending time to respond to such a
request would take time away from completing more HITs. To some extent, their
comments also make a case for the immediate financial gain of microwork – "an
hour of time" can make a difference in their earnings. In contrast, an
opportunity to share their stories appears to be uncompelling, at least in the
context of these specific requests and their intended usage. On various
occasions, we observed some Turkers sharing their prior experience with such
interview requests, suggesting that it is not an uncommon event in this
community.

> _“Some reporter shows up in this sub_[Reddit] _about once every 3-4 months
> soliciting workers for free info. If you want info about our work, make a
> HIT and pay us for our time. You’re literally coming to our forum and asking
> us to do the type of thing we do on MTurk, but for free.”_ [Turker6]
>
> _“Bummer, I just did a 70 minute interview with the New York Times for an
> article that’s coming out at the end of the week. I didn’t get paid for it
> though.”_ [Turker7]

In every thread we analyzed, we rarely found that Turkers showed enthusiasm or
expressed positive reactions to the interview requests. Instead, we
consistently observed negative experiences of Turkers as to these interview
requests in the forum; workers had done one but did not feel they were
adequately compensated, or they had seen these requests often and were
generally not satisfied with being asked to perform labor without compensation
for the time involved.

#### Distrust of media.

While compensation could be one cause for the rarity of positive reactions
from Turkers to these interview requests, some comments reveal more aspects of
their negative experiences with interviews:

> _“Twice I’ve done interviews with a journalist about MTurk and both times
> they misrepresented what I said. Never again”_ [Turker8]
>
> _"…I have yet to see an article that doesn’t quote people out of context
> and/or distort the facts entirely just to suit the title of the
> article."_[Turker9]
>
> _"With all due respect, the last few times there have been magazine
> articles, it has not gone well. There is always a promise to tell our side,
> and "give us exposure". The reality ends up being a smear campaign. Here are
> some examples…"_[Turker10]
>
> _"Be careful what you write my friend. With the declining state of print
> media and paid journalism you’ll be one of us before too long."_[Turker11]

These accounts reflect self-awareness of how Turkers are represented in media
narratives. In the elided portion of the quotation above, Turker10 listed
three articles from well-known media sources as examples of what they meant by
a promise to tell our side but ending up with being a smear campaign. In that
particular thread, the journalist followed up, referring to existing academic
work that demonstrates some visibility is needed for improving Turkers’
working condition. However, Turkers in that thread did not seem to appreciate
the work.

One Turker asserted that they would trust their own experience rather than a
theory written by others. Another Turker, more pointedly, alleged that both
academic work and journalist articles _utilized the lowest tier of workers to
create some sort of sob story that all mturk workers are being taken advantage
of_ , and these works are _poorly perceived in the turking community_.
Turker11 wrote with a sarcastic tone. We observe distrust toward the
journalists and researchers who post, due to Turkers being disappointed by
prior interviews and/or news articles and concerned with distortion and
misrepresentation.

In Turkers’ reactions to news articles we reviewed, this sentiment seems to be
salient: they tend to express a low expectation for any coverage up-front and
only react more positively when find the article does not paint a hopeless
picture of them. For instance, in reaction to recent coverage from a TV
channel, one Turker posted:

> _“At least her team did their study calling us Turkers.”_ [Turker12]

Turker12 here does not comment about the news coverage itself, but notes that
it is not that bad because they were at least being addressed in the right
way, showing no greater expectation for the reporting. Indeed, in some threads
of interview requests, when the journalist or researcher addressed Turkers
alternatively such as MTurks and Amazonians, Turkers reacted with a stronger
tone, trying to establish their identity as workers and sometimes Turkers.

> _“First things first: we’re not MTurks. MTurk is the platform. We are
> workers.”_ [Turker13]
>
> _"Mturks": the FIRST CLUE you’ve never even so much as looked at the
> platform, because if you did, you’d know we’re WORKERS (meaning, people).”_
> [Turker14]

In another discussion thread, where Turkers commented on a recent media
article about them, concerns expressed showed that workers were apprehensive
even before reading the article or became worried when they saw the headline
or the first few statements of the article solely emphasizing low pay.
However, after reading the full article, upon finding that the news story did
not just present a one-sided account of crowd work, workers often expressed
relief.

#### Fallacies of representation.

Another Turker had more specific comments in the aforementioned discussion
about the TV coverage of Turking, pointing to two potential fallacies in news
representation of Turkers: first, who from the Turker population was being
interviewed; second, whether the news producers had a formulated narrative
beforehand that could lead to selective reporting.

> _“The problem with the articles she cited is that they’re either
> interviewing people who are ridiculously new to the platform and just view
> it as "$5/day beer money" or whatever, or they’re stacked by people who have
> an agenda and seek to push a specific narrative…”_ [Turker15]

In the comments on different media articles, we found similar voices alleging
that there is a pre-set agenda in news reporting about Turkers, particularly
about their status of being underpaid and overworked and exploited. Some
Turkers in the forum expressed fatigue regarding this recurrent narrative.

> _“…the very studies that push the "underpaid and overworked and exploited"
> were all written with an agenda in mind. Instead of researching what would
> be a fair pay for turkers, some requesters just up and leave or post their
> work on other platforms.”_ [Turker16]
>
> _“…Most MTurk communities have seen this article rehashed time and time
> again, and we all knew what this was going to say long before it came out.”_
> [Turker17]

As we examined the original media coverage being discussed in our identified
forum threads, we also noticed common analogies repeatedly used in media
coverage to describe crowd work. For example, in one discussion thread,
Turkers appear in a media article [45] that refers to gig work as digital
slavery and crowd work as click farms. Turkers expressed their unease:

> _“I find the "slavery" analogies offensive…”_ [Turker18]

Under another article posted that frames crowd work platforms as virtual
sweatshop, one Turker replied sarcastically:

> _“"digital underclass" is the best description I’ve heard of turking so
> far.”_ [Turker19]

#### Anxiety about publicity.

When Turkers exhibit resistance to interview requests and passivity to media
coverage, they demonstrate unease with the possible or gained publicity. For
example, when expressing some dissatisfaction with some media reporting, a
Turker voiced clear opposition to such publicity:

> _“Below, you will find three examples of why this will be a hard pass from
> almost everyone who has turked for a while. We are fine without extra "help"
> or "publicity”_ [Turker20]

Turker20 then commented on three articles that have given unfavorable
publicity to Turkers. The reason that Turker20 held was that to portray
Turkers as exploited by digital platforms could turn task Requesters away
because crowd work would be perceived as unethical. However, Turker20 also
acknowledged that even in the case of positive publicity about Turking, it
would just lead to more workers coming to the platform, competing with
existing Turkers for limited task offerings. Similar voices abound:

> _“Please don’t give mturk any more publicity. There is a limited amount of
> work available, and if people learn about mturk and decide to join, it’s
> less money for us”_ [Turker21]
>
> _“Publicity really isn’t appreciated at this juncture. We’re just coming
> away from a rough NYT article and I think most turkers wish the 4th estate_
> [press and news media] _would forget about us for a little while.”_
> [Turker22]
>
> _“Talking to journalists is literally putting your income at risk for no
> reason.”_ [Turker23]

Essentially, Turkers would rather have their Turking opportunities secured,
instead of gaining visibility that could jeopardize their opportunities for
earning income. Their sentiment expresses risk aversion, since there is also
the possibility that negative publicity could drive current Turkers away from
the platform (thus fewer competitors for existing Turkers), and positive
publicity could bring more Requesters to the platform. Potential downsides
occupy their focus.

###  4.2 Diverse voices

#### The omitted diversity.

In one case when a journalist tried to recruit workers to interview, the
journalist even promised that this isn’t going to be another digital sweatshop
story. One worker still responded:

> _“The problem with these journalists is that they just go around and PM
> random users on this sub[reddit] and just get totally wildly different
> answers. Mturk is very subjective, some people are terrible at it and/or
> don[’]t use scripts and might only make $4 a day and meanwhile someone who
> really gets Mturk and/or uses scripts can make $40 a day in the same amount
> of time as someone who doesn[’]t use scripts.”_ [Turker24]

This quotation speaks of some reasons why Turkers are not represented properly
in media coverage. The experience of Turking varies wildly from person to
person. Excess attention given to a universal, powerless image of Turkers in
the media narratives hides some Turkers who are motivated self-starters, as
spoken of by a Turker in the forum. This image overshadows the fact that
microwork also requires skills, expertise, and perseverance. While existing
academic work has highlighted skills and strategies for Turking (e.g., [46,
16]), we see here, as one Turker put it:

> _“But I will say this, the majority of high earners usually don’t browse
> this sub or bother with this subreddit because they are too busy working on
> making $100+ a day and just see the people here as fools (for the lack of a
> better word).”_ [Turker25]

It could be possible that high-earning Turkers have less visibility in this
Subreddit, from which our analysis is drawn. In the same thread, another
Turker noted that success in Turking requires dedication and perseverance.
These qualities are for learning the nitty-gritty of the platform, closely
observing and acting with the dynamics of the task marketplace, and
strategically making use of Turking tools. Prior work [25, 46, 37] has noted
that there is a ladder of experience levels among Turkers, from novice to
expert. It is likely that those aspects are given far less attention in the
popular narratives about Turking [47].

#### Diverse motivations for Turking.

Apart from varying experience levels, motivations for Turking also vary
greatly. We observed that some Turkers indeed spoke out about their own
conditions and experiences. Consistent with the findings above, these were not
early responses to interview requests nor reactions to media articles. We saw
that after a post develops into a longer discussion, some Turkers begin to
open up more about their individual circumstances. In such accounts, across
threads, Turkers shared their diverse motivations for Turking. Some described
themselves as socially anxious, autistic, or lacking the competence to thrive
in a regular labor market. Turking was characterized as providing them with an
alternative mode of work. Some workers expressed appreciation for the
flexibility of working outside an office or simply the pleasure of incremental
achievements. Some Turkers were explicit about doing side gig for making use
of their idle time to earn guilt-free money to treat themselves.

> _“I don’t accept HITs that pay that low. I make around $10/hour and I use
> that money for my bubble tea habit, Christmas presents for family and
> generally being able to treat myself occasionally.”_ [Turker26]
>
> _“Sometimes when I drop my daughter off for her 3-4 hour volleyball
> practices, rather than go home and sit around I go to Starbucks and turk
> over a cup of coffee or two.”_ [Turker27]
>
> _“I’ve paid entirely for a high end camera with Turk money, and this has
> become my favorite hobby and returned priceless memories for my family._
> [Turker28]
>
> _“My ability to do Turking adds $400-$600 a month without taking me out of
> my primary role, which means nearly the same amount of income as her
> previous position with much more family time.”_ [Turker29]

In these posts, Turking sounds rewarding and enjoyable to Turkers. In
contrast, some workers report reliance on Turking for their primary income:

> _“I do it because I’m not healthy enough to handle a "real" job right now.
> Turking lets me work from home whenever I feel able and take as many breaks
> as I need. ”_ [Turker30]
>
> _“I am a mom of three children and am trying to bring in extra money with
> MTurk. I have a degree but am unable to work outside of the home because two
> of my children have significant special needs.”_ [Turker31]
>
> _“I’m physically not able to go out to work right now, but I still need food
> and shelter. I’m not officially disabled (no health insurance = no doctor)
> so I’m not getting any money from the government. I’m doing as many online
> jobs as I can right now, but those aren’t so plentiful. I turk to survive.”_
> [Turker32]

Prior work [13, 11, 17, 16] has documented a wide range of motivations for
Turking across nationality and geography. From these accounts we collected in
the forum, Turkers’ motivations and perceptions of Turking vary according to
their life conditions. One is naturally less likely to view Turking as
entertainment when it serves as one’s primary source of income.

#### Diverse career perspectives.

Turkers come to MTurk with different life conditions and attitudes. As we saw
in the Subreddit, some workers do not think Turking to be a job or a career,
bringing correspondingly lower expectations for the work. In contrast, other
dedicated Turkers in the forum contend that:

> _“I treat my work on here like a real job and I guess many other_[s] _do
> not.”_ [Turker33]
>
> _“If you look at it like a job…then you are getting "training" or learning
> what that "boss" wants. It isn’t worth it sometimes if there is only one
> hit…but if it’s something you can bank like you just mentioned, it’s worth
> it.”_ [Turker34]
>
> _“I don[’]t even try and argue with them anymore. More work for me and
> others who diligently read when they cry and complain.”_ [Turker35]

Turker33 thinks the view of Turking as a real job is not common among Turkers.
But Turker34 adopts a serious attitude toward Turking as a job and recognizes
the personal growth gained through Turking. One Turker from the same thread
comments that the learning curve is steep for crowd work while another affirms
that to make the platform work for oneself time, experience, and a certain
mindset are required. Turker35 expresses a very practical attitude that it is
not worthwhile to get upset or argue with task Requesters who reject their
work, while outrage regarding unfair rejection often features prominently in
popular press articles about Turking.

As we understand from these online discussions, recognizing Turking as formal
career experience seems helpful to breed resilience. These accounts suggest
that Turking is seen as a formal career option for some workers, in contrast
with the popular media portrayal of an exploited workforce.

##  5 Discussion

Based on our analysis of online discussion data, we have seen that in the
online Turker discourse community, Turkers tend to be disinclined to accept
interview requests. Specifically, Turkers are resistant to the idea of being
interviewed and sharing their stories, as they fear that publicity will turn
their work environment more competitive. They do not applaud media narratives
that portray them as a powerless and exploited underclass, even if their
working condition is not ideal.

In general, Turkers seem averse to media coverage adopting a more activist
tone as a preferable pathway to improve their working status. They express a
stronger attitude of self-determination and self-reliance regarding their
working decisions and whether or not anything needs to be rectified, and how
to go about it if so. They emphasize a simple desire for secure income
available from Turking.

Many researchers have discussed the invisibility of crowd workers (e.g., [31,
15]), and how this can lead to misrepresentation of the work involved and
increase precarity of work [48, 49]. Many academic efforts seek to probe into
the invisible crowd work and gain a deeper understanding. While online Turker
forums could serve as a channel for understanding these communities from
outside an opaque platform, there exist additional layers of friction to the
elicitation of Turkers’ voices and accounts. Turkers do not just easily tell
their stories.

In order to better understand Turkers’ experiences so that we might devise
methods to better support them and their work, we need to better understand
and respect their experiences and wishes. Well-meaning but naive or poorly-
executed interventions can cause harm and provoke anger from the very people
whom such interventions seek to help. While various studies have identifies
problems with worker invisibility, simply giving Turkers more exposure can
induce stress and resistance. Turkers largely do not seem to want the
attention that may disrupt their work and work environment, though they also
express a desire that the platform and income opportunities can be further
secured or bolstered.

One-sided media (or research) accounts of Turkers’ stories seem particularly
problematic, including distorting the public understanding and narrative
around crowd work. Other researchers have also noted this [41]. Many Turkers
in the online forum are aware of the simplistic caricatures of crowd work
illustrated in popular narratives. In contrast, their own accounts of Turking
are often a diverse and mixed bag of upsides and downsides, not necessarily
converging into one universal image. There is a clear gap between typical
media portrayals of crowd work and workers’ own perception of Turking. Use of
terminology such as digital slavery is particularly problematic as misleading,
demeaning, and racially charged. Turking itself requires special skills and
many HITs are also knowledge-intensive. Turkers’ work and professionalism
merit our respect.

When it is difficult to access an opaque platform and elicit Turkers’ voices,
journalist practices are particularly susceptible to the trap of parachute
journalism [50]. This refers to the production of quick, one-dimensional news
pieces that merely capture the clichés and stereotypes, often with stories
pre-defined before the reporter hits the ground in the locale to be reported
on. Some conditions, such as insufficiency of local knowledge and tight
deadlines, provide fertile ground for parachute journalism. Rather than
questioning professional ethics, we suggest more attention be given to the
nature of crowd work as a complex sociotechnical issue. Less visible online
work requires updated practices to learn about its operation in depth. Merely
interviewing a few crowd workers is unlikely to capture the rich diversity of
lives and lived experiences surrounding crowd work. Several times in worker
discussions we saw a critique of whether a journalist’s experience on AMT for
a couple of weeks is sufficient to understand Turking themselves, let alone
depict it accurately to their readers.

When we write Turkers, it is tempting to assume this name envelops a
homogeneous group, rather than recognizing it as an umbrella term encompassing
a rich diversity of underlying humanity and experiences. Beyond well-known
geographic diversity, there is diversity in a multitude of other respects,
including living conditions, motivations, valuation of Turking, skill and
experience levels, and career visions [51, 52]. Consequently, it is
challenging to effectively support this diverse workforce via any one-size-
fits-all intervention, policy, or regulation.

Because our findings are drawn from online observations in Turker forums and
their discussion data, our analysis is limited by that observed data. We miss
the voices of Turkers who do not post in online forums, the forum we selected,
or in other threads excluded from our sample. That said, our findings
complement the prior research work by shedding further light upon _why_
understanding and representing crowd workers is so difficult. Their size,
scale, diversity, invisibility, and resistance to publicity make it
particularly difficult to access them and risk bias-prone representations.
These constitute clear challenges to recognize in pursuing a worker-centered
agenda to support crowd work. It is also critical to strive to understand the
constituency of Turkers we wish to aid in such work.

For a worker-centered research agenda, what approaches would best enable us to
understand the shifting and diverse population of Turkers? How should we
respond to the invisibility of the workforce when they are resistant to
publicity? How could we achieve a worker-centered approach to crowd work when
it is inherently hard to realize so with traditional approaches? The voices of
Turkers we have heard lead us to these questions to help guide future
research.

## Acknowledgments

We thank all those who provided feedback to us in preparing this manuscript.
This work was supported in part by Good
Systems555https://goodsystems.utexas.edu, a UT Austin Grand Challenge to
develop responsible AI technologies. Matthew Lease is also supported as an
Amazon Scholar. Our opinions are entirely our own and do not speak to those of
the sponsoring agencies.

## References

  * [1] Jeff Barr and Luis Felipe Cabrera.  Ai gets a brain.  Queue, 4(4):24–29, 2006.
  * [2] Lilly Irani.  The cultural work of microwork.  New Media & Society, 17(5):720–739, 2015.
  * [3] Aaron Smith.  The Gig Economy: Work, Online Selling and Home Sharing.  Technical report, Pew Research Center, November 2016.
  * [4] Chris Callison-Burch and Mark Dredze.  Creating Speech and Language Data with Amazon’s Mechanical Turk.  In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT ’10, pages 1–12, Los Angeles, California, June 2010. Association for Computational Linguistics.
  * [5] Stefanie Nowak and Stefan Rüger.  How Reliable are Annotations via Crowdsourcing: a Study about Inter-annotator Agreement for Multi-label Image Annotation.  In Proceedings of the international conference on Multimedia information retrieval, MIR ’10, pages 557–566, Philadelphia, Pennsylvania, USA, March 2010. Association for Computing Machinery.
  * [6] Catherine Grady and Matthew Lease.  Crowdsourcing document relevance assessment with mechanical turk.  In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s mechanical turk, pages 172–179. Association for Computational Linguistics, 2010.
  * [7] Gabriele Paolacci, Jesse Chandler, and Panagiotis G. Ipeirotis.  Running Experiments on Amazon Mechanical Turk.  SSRN Scholarly Paper ID 1626226, Social Science Research Network, Rochester, NY, June 2010.
  * [8] Aniket Kittur, Ed H. Chi, and Bongwon Suh.  Crowdsourcing User Studies with Mechanical Turk.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’08, pages 453–456, Florence, Italy, April 2008\. Association for Computing Machinery.
  * [9] Djellel Difallah, Elena Filatova, and Panos Ipeirotis.  Demographics and Dynamics of Mechanical Turk Workers.  In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM ’18, pages 135–143, Marina Del Rey, CA, USA, February 2018. Association for Computing Machinery.
  * [10] M Six Silberman, Lilly Irani, and Joel Ross.  Ethics and tactics of professional crowdwork.  XRDS: Crossroads, The ACM Magazine for Students, 17(2):39–43, 2010\.
  * [11] Panagiotis G Ipeirotis.  Demographics of mechanical turk.  2010\.
  * [12] Joel Ross, Lilly Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson.  Who are the Crowdworkers? Shifting Demographics in Mechanical Turk.  In CHI ’10 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’10, pages 2863–2872, Atlanta, Georgia, USA, April 2010. Association for Computing Machinery.
  * [13] Judd Antin and Aaron Shaw.  Social Desirability Bias and Self-reports of Motivation: a Study of Amazon Mechanical Turk in the US and India.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12, pages 2925–2934, Austin, Texas, USA, May 2012. Association for Computing Machinery.
  * [14] Robin Brewer, Meredith Ringel Morris, and Anne Marie Piper.  "Why would anybody do this?": Understanding Older Adults’ Motivations and Challenges in Crowd Work.  In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, pages 2246–2257, San Jose, California, USA, May 2016. Association for Computing Machinery.
  * [15] Mary L. Gray and Siddharth Suri.  Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass.  Houghton Mifflin Harcourt, 2019.
  * [16] Toni Kaplan, Susumu Saito, Kotaro Hara, and Jeffrey P. Bigham.  Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers.  In Sixth AAAI Conference on Human Computation and Crowdsourcing, June 2018.
  * [17] Nicolas Kaufmann, Thimo Schulze, and Daniel Veit.  More than fun and money. worker motivation in crowdsourcing-a study on mechanical turk.  In AMCIS, volume 11, pages 1–11. Detroit, Michigan, USA, 2011.
  * [18] David Martin, Benjamin V. Hanrahan, Jacki O’Neill, and Neha Gupta.  Being a Turker.  In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, CSCW ’14, pages 224–235, Baltimore, Maryland, USA, February 2014. Association for Computing Machinery.
  * [19] Lydia B. Chilton, John J. Horton, Robert C. Miller, and Shiri Azenkot.  Task Search in a Human Computation Market.  In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP ’10, pages 1–9, Washington DC, July 2010. Association for Computing Machinery.
  * [20] John Joseph Horton and Lydia B. Chilton.  The Labor Economics of Paid Crowdsourcing.  In Proceedings of the 11th ACM conference on Electronic commerce, EC ’10, pages 209–218, Cambridge, Massachusetts, USA, June 2010\. Association for Computing Machinery.
  * [21] Brian McInnis, Dan Cosley, Chaebong Nam, and Gilly Leshed.  Taking a HIT: Designing around Rejection, Mistrust, Risk, and Workers’ Experiences in Amazon Mechanical Turk.  In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, pages 2271–2282, San Jose, California, USA, May 2016. Association for Computing Machinery.
  * [22] M Six Silberman, Bill Tomlinson, Rochelle LaPlante, Joel Ross, Lilly Irani, and Andrew Zaldivar.  Responsible research with crowds: pay crowdworkers at least minimum wage.  Communications of the ACM, 61(3):39–41, 2018.
  * [23] Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P. Bigham.  A Data-Driven Analysis of Workers’ Earnings on Amazon Mechanical Turk.  In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, pages 1–14, Montreal QC, Canada, April 2018. Association for Computing Machinery.
  * [24] Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Benjamin V. Hanrahan, Jeffrey P. Bigham, and Chris Callison-Burch.  Worker Demographics and Earnings on Amazon Mechanical Turk: An Exploratory Analysis.  In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–6, Glasgow Scotland Uk, May 2019. ACM.
  * [25] Neha Gupta, David Martin, Benjamin V. Hanrahan, and Jacki O’Neill.  Turk-Life in India.  In Proceedings of the 18th International Conference on Supporting Group Work, GROUP ’14, pages 1–11, Sanibel Island, Florida, USA, November 2014. Association for Computing Machinery.
  * [26] Mary L. Gray, Siddharth Suri, Syed Shoaib Ali, and Deepti Kulkarni.  The Crowd is a Collaborative Network.  In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW ’16, pages 134–147, San Francisco, California, USA, February 2016. Association for Computing Machinery.
  * [27] David Martin, Jacki O’Neill, Neha Gupta, and Benjamin V. Hanrahan.  Turking in a Global Labour Market.  Computer Supported Cooperative Work (CSCW), 25(1):39–77, February 2016.
  * [28] Benjamin B. Bederson and Alexander J. Quinn.  Web Workers Unite! Addressing Challenges of Online Laborers.  In CHI ’11 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’11, pages 97–106, Vancouver, BC, Canada, May 2011. Association for Computing Machinery.
  * [29] Niloufar Salehi, Lilly C. Irani, Michael S. Bernstein, Ali Alkhatib, Eva Ogbe, Kristy Milland, and Clickhappier.  We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers.  In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15, pages 1621–1630, Seoul, Republic of Korea, April 2015. Association for Computing Machinery.
  * [30] Mark E. Whiting, Grant Hugh, and Michael S. Bernstein.  Fair Work: Crowd Work Minimum Wage with One Line of Code.  Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 7(1):197–206, October 2019.  Number: 1.
  * [31] Lilly C. Irani and M. Six Silberman.  Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’13, pages 611–620, Paris, France, April 2013\. Association for Computing Machinery.
  * [32] Geoff Cooper and John Bowers.  Representing the User: Notes on the Disciplinary Rhetoric of Human-Computer Interaction.  In The Social and Interactional Dimensions of Human-Computer Interfaces, pages 48–66. Cambridge University Press, May 1995.
  * [33] Donna J. Haraway and Thyrza Goodeve.  Modest_witness@Second_millenium.  In Modest_Witness@Second_Millennium. FemaleMan_Meets_OncoMouse: Feminism and Technoscience. Routledge, 1997\.
  * [34] Lilly C. Irani and M. Six Silberman.  Stories We Tell About Labor: Turkopticon and the Trouble with "Design".  In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, pages 4573–4586, San Jose, California, USA, May 2016. Association for Computing Machinery.
  * [35] Guidelines for Academic Requesters | We Are Dynamo Wiki | Fandom, September 2017.
  * [36] Akash Mankar, Riddhi J. Shah, and Matthew Lease.  Design Activism for Minimum Wage Crowd Work.  In 5th AAAI Conference on Human Computation and Crowdsourcing (HCOMP): Works-in-Progress Track, 2017.  See extended technical report: arXiv 1706.10097.
  * [37] Benjamin V Hanrahan, Anita Chen, JiaHua Ma, Ning F Ma, Anna Squicciarini, and Saiph Savage.  The expertise involved in deciding which hits are worth doing on amazon mechanical turk.  Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1–23, 2021.
  * [38] Susumu Saito, Chun-Wei Chiang, Saiph Savage, Teppei Nakano, Tetsunori Kobayashi, and Jeffrey Bigham.  Predicting the working time of microtasks based on workers’ perception of prediction errors.  Human Computation, 6:192–219, 2019.
  * [39] Carlos Toxtli and Saiph Savage.  Meta-gig: Empowering anyone to create crowd marketplaces.  Avances en Interacción Humano-Computadora, 1:11–19, 2020.
  * [40] Claudia Flores-Saviaga, Yuwen Li, Benjamin Hanrahan, Jeffrey Bigham, and Saiph Savage.  The challenges of crowd workers in rural and urban america.  In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 159–162, 2020.
  * [41] Aaron J Moss, Cheskie Rosenzweig, Jonathan Robinson, and Leib Litman.  Is it ethical to use mechanical turk for behavioral research? relevant data from a representative survey of mturk participants and wages.  2020\.  https://psyarxiv.com/jbc9d.
  * [42] James P Spradley.  Participant observation.  Waveland Press, 2016.
  * [43] Virginia Braun and Victoria Clarke.  Using Thematic Analysis in Psychology.  Qualitative Research in Psychology, 3(2):77–101, January 2006.
  * [44] Virginia Braun and Victoria Clarke.  Thematic analysis. American Psychological Association, 2012.
  * [45] Ming Lim.  Why Many Click Farm Jobs should be Understood as Digital Slavery, January 2018.
  * [46] Saiph Savage, Chun Wei Chiang, Susumu Saito, Carlos Toxtli, and Jeffrey Bigham.  Becoming the Super Turker:Increasing Wages via a Strategy from High Earning Workers.  In Proceedings of The Web Conference 2020, pages 1241–1252, Taipei Taiwan, April 2020. ACM.
  * [47] Chris.  Writer who never learned to drive works for uber. makes $0.97/hr.  TurkerView Blog, 2019.  https://blog.turkerview.com/writer-who-never-learned-to-drive-works-for-uber/. November 18.
  * [48] Lucy Suchman.  Making Work Visible.  Communications of the ACM, 38(9):56–64, September 1995.
  * [49] Susan Leigh Star and Anselm Strauss.  Layers of Silence, Arenas of Voice: The Ecology of Visible and Invisible Work.  Computer Supported Cooperative Work (CSCW), 8(1):9–30, March 1999\.
  * [50] Sharyn Wizda.  Parachute journalism.  American Journalism Review, 19(6):40–45, 1997.
  * [51] Andy Baio.  The Faces of Mechanical Turk, 2008.
  * [52] Jonas Oppenlaender, Aku Visuri, Kristy Milland, Panos Ipeirotis, and Simo Hosio.  What do crowd workers think about creative work?  arXiv preprint arXiv:2002.10887, 2020.