# Fire Dragon and Unicorn Princess

Gender Stereotypes and Children’s Products in Search Engine Responses

Amifa Raj  People and Information Research Team
Boise State UniversityBoiseIdahoUSA83725-2055<EMAIL_ADDRESS>and
Michael D. Ekstrand  People and Information Research Team
Boise State UniversityBoiseIdahoUSA83725-2055<EMAIL_ADDRESS>
(2022)

###### Abstract.

Search engines in e-commerce settings allow users to search, browse, and
select items from a wide range of products available online including
children’s items. Children’s products such as toys, books, and learning
materials often have stereotype-based gender associations. Both academic
research and public campaigns are working to promote stereotype-free childhood
development. However, to date, e-commerce search engines have not received as
much attention as physical stores, product design, or marketing as a potential
channel of gender stereotypes. To fill this gap, in this paper, we study the
manifestations of gender stereotypes in e-commerce sites when responding to
queries related to children’s products by exploring query suggestions and
search results. We have three primary contributions. First, we provide an
aggregated list of children’s products with associated gender stereotypes from
the existing body of research. Second, we provide preliminary methods for
identifying and quantifying gender stereotypes in system’s responses. Third,
to show the importance of attending this problem, we identify the existence of
gender stereotypes in query suggestions and search results across multiple
e-commerce sites.

information retrieval, gender stereotype, search results, query suggestions,
children, e-commerce

††copyright: rightsretained††conference: ACM SIGIR Workshop on eCommerce; July
15, 2022; Madrid, Spain††journalyear: 2022

##  1\. Introduction

Information retrieval (IR) systems such as search engines for e-commerce
settings serve a wide range of users satisfying their various information
needs for different context. Further, beyond their ability to meet a user’s
information need, there are number of potential social concerns raised by
these systems, and a growing body of research is actively studying various
aspects of the ethical (Milano et al., 2020; Urman et al., 2021), bias
(Otterbacher, 2018), and fairness (Ekstrand et al., 2021) challenges in IR
systems. However, the impact of these ethical issues on children has been
overlooked in the corresponding research. We are concerned with an important
issue that has not yet seen much study: the impact of representational harms
(Kate, 2017) in search engine responses, particularly the way gender
stereotypes may be reflected to children. Noble (2018) highlighted how search
engines can reinforce negative gender, racial, and intersectional stereotypes
when used by children, and the risk such exposure can cause; in this paper, we
attempt to measure the presence and extent of gender stereotypes for
children’s products in search systems of e-commerce settings.

Ellemers (2018) describes gender stereotypes as shared (often implicit)
beliefs and social expectations associated with people based on their gender.
One way this manifests for children is by categorizing toys based on the
gender of the child expected to play with them, reinforcing social
expectations and encouraging people to believe that products can (and should)
be differentiated based on gender (Liben et al., 2018). These unnecessary
gender associations appear in marketing, packaging, and advertisement of
children’s products through a variety of means, including color, labeling, and
images (Berenbaum, 2018; Fulcher and Hayes, 2018). Public-interest campaigns
have documented the presence of gender stereotypes in the promotion of
children’s products and the negative implications of this practice (Leanne,
2015; Culhane and Bazeley, 2019); psychology research has also studied the
existence and impact of gender stereotypes in children’s products (Kneeskern
and Reeder, 2020; Seitz et al., 2020; Skočajić et al., 2020) showing the
importance of considering this problem.

Most of the efforts on this to date have focused on marketing and store
design, but these are not the only channels through which stereotypes may be
expressed. Since various e-commerce systems are frequently used to advertise,
sell, and market children’s products, the product marketing and store design
itself are not the only channels through which stereotypes may be expressed:
the search engines in those e-commerce sites may also replicate and reinforce
gender stereotypes associated with children’s products.

It is important to investigate whether and how these systems may perpetuate or
reinforce gender stereotypes to understand the potential impact of the
existence of gender stereotypes in this source and thus developing strategies
to measure and mitigate the gender stereotypes associated with children’s
products. Baker and Potts (2013) and Noble (2018) showed that search engines
often stereotypically associate gender in their responses. We previously
argued the need to consider the role of IR in replicating and reinforcing
gender stereotypes regarding children’s products through additional examples
specifically focusing on learning environments (Raj et al., 2021). Building on
that agenda, we provide initial quantitative results on detecting and
exploring the presence of gender stereotypes associated with children’s
products in e-commerce search systems to further highlight the need to
investigate the role of IR in reinforcing gender stereotypes, particularly for
search tasks that affect children (either because children are directly using
the system, or because adults are using it to find information and products
for children).

Our goal is to identify how gender stereotypes may appear in _query
suggestions_ and _search results_ when searching for children’s products in
e-commerce settings. We examined multiple e-commerce systems and analyzed
their responses for associations of gender with children’s products, based on
a collection of commonly gender-stereotyped children’s products from previous
research.

We have three main contributions:

  * •

We provide an aggregated list of children’s products with associated gender
stereotypes from existing research.

  * •

We provide a simple and explainable preliminary methods to quantify gender-
product associations while exploring the existence and persistence of gender
stereotypes associated with children’s products in query suggestions and
search results in e-commerce settings.

  * •

We identify the potential existence of gender stereotypes in query suggestions
and search results of search engines, across multiple e-commerce systems,
showing the importance of attending to this problem.

This work is not primarily an audit study to make normative claims about any
particular e-commerce systems, although we hope developers of such systems
will attend to our results. We focus on providing empirical support for the
existence of the problem across the e-commerce search landscape in general as
a prompt for discussion and further research.

##  2\. Background

Our work draws primarily from three streams: gender stereotypes in children’s
products, gender stereotypes in IR, and IR for children.

###  2.1. Gender Stereotype in Children’s Products

The term “stereotype” refers to the concept of categorizing people into groups
and associating popular beliefs or common patterns with those groups
(Cauteruccio et al., 2020). Gender stereotypes are common beliefs and social
expectations associated with particular genders (Ellemers, 2018; Cauteruccio
et al., 2020).

Gender stereotypes have long been associated with children’s products; these
associations persist in society and are reinforced by the marketing,
packaging, and advertising of children’s products (Auster and Mansbach, 2012).
The trend of assuming gender appropriateness for toys leads to gender
stereotypes associated with them. Cherney (2018) and Blakemore and Centers
(2005) showed that characteristics such as attractive, shiny, domestic,
nurturing, fashion-oriented, pretend-play, and pink or pastel colors are
commonly associated with “toys for girls”, while active, adventurous, violent,
aggressive, spatial, movement-based, STEM, competitive, somewhat dangerous,
and black or dark colored are associated with ”toys for boys”. Retailers and
media reflect this categorization and further reinforce them through marketing
(Auster and Mansbach, 2012), advertising (Azmi et al., 2021), and packaging
(Souza et al., 2021). Children are exposed to this concept of gender-based
toys through their social interactions, including parents, peers, and signals
in society at large (Freeman, 2007; Kollmayer et al., 2018; Eisen et al.,
2021).

Weisgram and Dinella (2018) explored various aspects of gender stereotypes
associated with children’s products, analyzing the characteristics of gendered
products and the causes and consequences of gender-typed playing. Early
childhood exposure to gender stereotypes through playing or engaging with
gender-stereotyped products has long term impact on physical, social, and
cognitive development including future ability, skills, personality, social
interaction, perception towards gender roles, and career interests (Fulcher
and Coyle, 2018; Dinella and Weisgram, 2018; Fulcher and Hayes, 2018; Spinner
et al., 2018; Weisgram et al., 2014).

In addition to academic research on stereotypes and their effects, consumer
campaigns have worked to break gender-stereotyped toy practices. Efforts such
as Let Toys be Toys111https://www.lettoysbetoys.org.uk/ and Smash
Stereotypes222https://www.fawcettsociety.org.uk/smashstereotypes are working
to challenge gender stereotypes in children’s products with the goal of
providing children with a stereotype-free childhood. These campaigns have had
impact; the popular retailer Target (Dinella and Weisgram, 2018) and
manufacturer LEGO (Russell, 2021) have both decided to eliminate gender-based
categorization of their products. California has also enacted a law requiring
gender-neutral sections in stores to reduce gender-based discrimination (BEAM,
2021).

###  2.2. Gender Stereotypes in IR

Ethical and social concerns, such as fairness, receive significant research
interest in IR (Ekstrand et al., 2019; Gao and Shah, 2021; Urman et al.,
2021). Several studies have specifically looked to identify, assess, and
mitigate gender-based social biases in IR (Raj et al., 2020; Makhortykh et
al., 2021; Epps-Darling et al., 2020; Ekstrand and Kluver, 2021). Much of this
work, however, focuses on distributional harms: ensuring that providers and
users of different genders receive equitable benefits from the system.
Representational harms — harms arising from how the system represents users or
items, either internally or in its results — of IR is still under-studied;
replicating and reinforcing gender stereotypes is one such harm.

In AI and machine learning more broadly, Abbasi et al. (2019) studied
stereotypes as representation harm in machine learning pipelines and Ahn et
al. (2022) showed the effect of gender stereotypes in AI-recommended products
by investigating human interaction with gendered AI agents. In IR, Kay et al.
(2015) showed the existence of gender stereotypes in image search results for
occupations, as evaluated by human participants, and Magno et al. (2016) found
that image results may impose certain attractiveness stereotypes depending on
the language used to formulate queries. This work has not considered the
presence of gender stereotypes in textual representations, or in product
results of search engines in e-commerce settings. The closest work to our own
is that of Fabris et al. (2020), who looked at gender stereotypes in retrieved
documents of search results and proposed a Gender Stereotype Reinforcement
measure leveraging Word2Vec’s (Mikolov et al., 2013) tendency to reflect
gender stereotypes in word embeddings. Fabris et al. evaluated well-known IR
algorithms on the Robust04 TREC dataset (Voorhees et al., 2004), detecting
stereotypes in a controlled environment instead of production systems.

While there is some progress on recognizing the presence of stereotypes in IR
systems (Noble, 2018; Baker and Potts, 2013; Roy and Ayalon, 2020; Sweeney,
2013), there hasn’t been enough work that can help to systematically detect,
assess, and mitigate gender stereotypes in IR. The work to date has also
focused on the system’s primary results, such as search results in response to
a query; how various phases across an IR workflow (such as search results and
query suggestions) replicate and exhibit gender stereotypes is still
unexplored, as is the the role of IR in replicating and reinforcing gender
stereotypes associated with specific problem settings like searching for
children’s products has not received attention to our knowledge.

###  2.3. IR for Children

Most research on IR, including research on its ethical and social dimensions,
usually assume a “default” user or society member, often implicitly assumed to
be an adult. There is a line of work on building better web environments for
children, particularly in search; this research often focuses on explicit
content concerns such as inappropriate language or adult content (Madrazo
Azpiazu et al., 2018; Alghowinem, 2018), or on better meeting children’s
information needs (Dragovic et al., 2016). Connecting the work on ethical and
social aspects of IR, such as fairness (Ekstrand et al., 2021), with the
particular needs and experiences of children is a significant gap in the
literature. Noble (2018) showed how search systems can perpetuate negative
stereotypes in a way that impacts children, but most work building on her
argument has focused on the reproduction of harmful societal ideas, not on
their particular impacts on children. Thus, the role of search systems in
manifesting harmful expectations and beliefs to children through their
response is still overlooked; filling this gap will help achieve the goal of
building safer search environments for children.

In this work, we draw our attention to two important issues that have not yet
seen much study: considering the impact (and resulting ethical considerations)
of search systems specifically on children; and studying representational
harms in their response, particularly the way gender stereotypes may be
reflected.

##  3\. Data and Methods

To identify the existence and persistence of gender stereotypes associated
with children’s toys in search systems in e-commerce settings, we explore
query suggestions and search results quantifying the gender associations that
manifest in each of these stages. We seek simple and explainable methods to
provide a first approach in quantifying gender-product associations to
demonstrate their existence and serve as a foundation for future work that
will refine and further develop relevant measurements.

To conduct our experiments, we explored two popular US e-commerce sites:
Amazon333https://www.amazon.com/, because they are the most widely-used
e-commerce site in the United States (Stephanie, 2021) and
Target444https://www.target.com/, because they have taken action to end
gendered categorization of toys (Murnen, 2018; Dinella and Weisgram, 2018),
and we want to see if this is reflected in search engine responses.

###  3.1. Documented Toy Gender Stereotypes

To identify the presence of gender stereotypes associated with children’s
products in a systematic way that connects to existing research and advocacy,
we needed to identify items that are often have stereotyped gender
associations. Existing research and advocacy campaigns provide lists of
commonly-held toy-gender associations, categorizing toys as being targeted
towards boys or girls; we used four of these lists to build our stereotyped
toy sets: Richardson and Simpson (1982), Blakemore and Centers (2005), Murnen
et al. (2016), and Smash Stereotypes by Fawcett (Culhane and Bazeley, 2019).
We combined the toys listed for each gender category by each of these studies
and reports and removed duplicates to obtain lists of boy-associated, girl-
associated, and neutral toys; Table 1 shows the resulting full toy lists.

Table 1. List of toys and gender associations from previous papers and campaigns. Boys |  toy vehicles; military toys; race cars; outer space toys; construction toys; car toys; video games; building blocks; dinosaur toys; lego cars; lego toys; depots; machines; doll-humanoid; action figures; gi joe action figure; spider-man; guitar; castle tent; microscope; weather forecasting toy; bug collection set; barn; grill; toy guns; volcano creator; pokemon cards; space station toys; basketball hoop; gears; train set; lincoln logs; police station toy; airport toy; police officer gear; dragon ball z; superhero costume; helicopter; racetrack; remote controlled cars; football; hockey stick; sword toy; miniature guns; truck set; sully costume; woody valentine; spider man costume; batman costume; ninja costume; wwe action figure; construction toys; war toys; educational material; computer games; spaceship lego; firefighter gear; soldier toys; space station; science project kit; toy rocket; soccer ball; toy robots; toy drones; monster trucks; nerf guns
---|---
Girls |  doll; domestic toys; educational art; clothes; dollhouses; clothing accessories; doll accessories; furnishing; ballerina costume; barbie costume; barbie doll; barbie jeep; play makeup; bratz doll; jewelry; my little pony; tea set; vanity set; princess costume; baby doll; beads; easy bake oven; baby doll stroller; sewing machine; pink ice skates; ken doll; toy kitchen; broom; beanie baby bear; toy vacuum cleaner; veterinarian kit; kitchen set; stuffed animals; plush toys; unicorn; sparkly toys; disney doll; baking kit; painting kit; dora doll; power ranger valentine; dorothy costume; minnie mouse; monster high doll; monster high costume; fashion doll; tiara; craft toys; beauty products; tea party set; pink doll; cupcake maker; makeup kit; fashion dolls; crochet kit; playhouse; pink ice skates; princess sword; doctors kit; skin care kit; barbie furniture set
Neutral |  toy animals; educational teaching; musical games; games; books; live animals; candy land; winnie the pooh; karaoke machine; elmo; gardening tools; crayons; doctor kit; tricycle; play-doh; leappad; trampoline; swing set; mr potato head; spongebob square pants; math flash card; wagon; bus; tree house; wood blocks; harry potter books; scooter; drum set; puzzles; board games; rock painting

(a) Query suggestion for “cars for” in Amazon

(b) Search results for “cars for boys” from Amazon

(c) Search results for “cars for girls” from Amazon

Figure 1. Query suggestions and search results for cars

###  3.2. Query suggestions

Many search engines provide _query suggestions_ to aid writing query process.
The suggested query completions sometimes include gender markers; for example,
figure 1(a) shows an e-commerce site suggesting “cars for boys” as a
completion of “cars for”. This is one way the e-commerce search systems may
replicate existing gender stereotypes: the presence of the gendered term
reflects an association between the initial query and gender, typically
learned from underlying data. It may further reinforce those gender
stereotypes by nudging users to select the gendered query.

We presented each item in Table 1 as a query to Amazon and Target search
systems and recorded the completions it suggested using the fake-
useragent555https://pypi.org/project/fake-useragent/ package. We then counted
how often each system suggested a completion containing a gender to measure
the gender association that system inferred for each product. This setup
allows us to identify if search systems associate target gender with items
through query suggestions and whether those associations replicate previously-
documented gender stereotypes.

###  3.3. Search Results

When a search system retrieves results, it typically presents them in a ranked
list sorted by their estimated relevance to the query (Schütze et al., 2008).
One way the system may reflect gender associations and stereotypes is in its
response to gendered queries: when a query explicitly mentions a gender, the
system may retrieve a different set of relevant results than it would with a
different gender, reflecting particular gender associations back to its users.
In addition to the products themselves, gender stereotypes can manifest in
search results through various characteristics such as colors, keywords,
terms, and features associated with the retrieved products. Figure 1(b) and
figure 1(c) show top results for the queries “cars for boys” and “cars for
girls”, respectively showing difference in colors, products, features, and
keywords with the change of gender in the query. Thus search results may
reinforce and propagate gender stereotypes by reflecting unnecessary gender
association with children’s products.

We collected search results from Amazon and Target to explore their role in
replicating and propagating gender stereotypes in children’s products through
the search results themselves. For each item in Table 1, we used
Selenium666https://www.selenium.dev/ to issue both gender-specific and gender-
neutral queries, render the search result pages, and collect the titles of the
retrieved products.

To explore the differences in response to gender, we observe the difference in
keywords associated with retrieved products when we issue a query with
explicitly mentioning gender (gender-specific queries), either one of the
items from Table 1 (e.g. “truck set for girls”) or a generic query (e.g. “toys
for boys”). For each query, we extracted frequent keywords from search results
and identified unique terms that are commonly associated with products
retrieved given a specific gender. This strategy let us explore how systems
replicate social stereotypes while retrieving gender-specific results.

To identify specific toy-gender associations in search results, we
investigated the similarity between gender-neutral and gender-specific (gender
mentioned in query) retrieved result sets for each item; if the neutral search
results closely match with a specific gendered search results, we identify
that product to be associated with that gender. For example, if the search
results of the query “dollhouse” have more similarity with search results of
“dollhouse for girls” than “dollhouse for boys”, we assume the item
(dollhouse) is to be associated with the female gender.

##  4\. Empirical Results

We now turn to the results of this exploratory analysis and to that end, we
answered two primary research questions:

RQ1.:

Do query suggestions reflect gender stereotypes associated with children’s
products?

RQ2.:

Do search results manifest gender stereotypes associated with children’s
products?

###  4.1. Query Suggestions

Applying the collection strategy described in Sec. 3.2 to the two e-commerce
systems we studied yielded a total of 334 query suggestion sets. For a given
system and a query, we collected the set of suggested queries. We then
computed a girl-association score by counting the number of times “girl” or
“girls” appears in the query suggestions (e.g. by expanding “truck” to “truck
for girls”); likewise we compute boy-association score by looking for “boy”
and “boys”. We used the difference between these two scores in determining the
system associated gender label for each item. Our investigation into this
first question is in the form of three sub-questions.

RQ1.a. Do e-commerce search systems associate gender with children’s products
in query suggestions?

To observe either system’s tendency of attaching gender with children’s
products, we computed the number of items that were associated with genders by
the systems while suggesting queries. Figure 2(a) shows that number of items
each system associated with each gender. We first look at how likely the
system is to associate a gender at all; Amazon had the highest tendency of
attaching a gender to items through its query suggestions.

RQ1.b. Do e-commerce search systems replicate stereotypes associated with
children’s products through query suggestions?

After identifying the tendency of associating gender with items through query
suggestion, we wanted to explore how likely these systems reflect previously-
documented labels for the items. With that goal, we examined the individual
lists of items with gender association derived from each system.

To quantify how each system _replicates_ gender stereotypes through its query
suggestions, we compared the gender associations in its query associations
with the previously-documented stereotypes. For each system, we calculated the
Jaccard similarity between sets of items the system associates with a gender
and the set of items for that gender from the prior research (Table 1). Figure
2(b) shows the similarity score for each system. Both system’s propagation of
stereotype was statistically significantly greater than expected by chance
($p<10^{-5}$ with a one-tailed randomization test777Our sets of items are not
representative or random samples of the space of possible items, so most
statistical tests are not appropriate; the randomization test’s null
hypothesis that the labels are independent of other item attributes is more
appropriate for this analysis. For each system, we shuffled the system-
generated gender label of items and calculated the similarity score between
the item list for a gender and the pre-documented item list for that gender,
thus simulating the distribution of similarities under the null hypothesis.).
This provides evidence that these systems are replicating the gender
stereotype associations of our toy list in their query suggestions.

Our results show that query suggestions appear to reproduce previously-
documented gender stereotypes to some extent, in that the stereotyped gender
is more likely to appear in query suggestions when searching for that item.

RQ1.c Does the prevalence of gender stereotypes vary between query suggestions
of different systems?

After observing the overall tendency of e-commerce search system to replicate
gender stereotypes through query suggestions, we now turn to whether the
systems differ in their responses.

From figure 2(b), we observe that Amazon and Target have similar query
suggestion rates (fraction of items that have suggestions) but differ in their
tendency of associating gender with items at all, in addition to differing in
their replication of previous gender stereotypes through query suggestions.
This difference in their traits is statistically significantly greater than
expected by chance ($p<0.001$ with a two-tailed randomization test888We
randomly swapped the system-generated gender labels among systems. Then for
each system, we calculated the similarity score between the item list for a
gender and the pre-documented item list for that gender and computed the
difference between the similarity score of two systems). This implies that
these systems have the tendency of replicating gender stereotypes through
query suggestions.

(a) Item gender associations.

(b) Similarity between system derived list and stereotyped toy list.

Figure 2. Persistence of previously-documented gender stereotype associated
with children’s products across e-commerce query suggestions

Key Findings: We have explored two e-commerce systems to identify if and how
gender stereotypes manifest through query suggestions. Moreover, how the
prevalence of gender stereotypes vary across these systems.

From our analysis of RQ1, we have the following findings:

  * •

E-commerce search systems frequently target gender for children’s items
through query suggestions.

  * •

The overall response (query suggestions) of all systems imply that gender
stereotypes associated with children’s products persist in search systems in
e-commerce settings to at least some extent.

  * •

The tendency of associating target gender with children’s items vary across
the selected systems.

  * •

The tendency of reflecting previously-documented gender stereotypes associated
with children’s products through query suggestions vary between Amazon and
Target.

Table 2. Top-10 most frequent unique words extracted from gender-specific search results | Boys Only | Girls Only
---|---|---
| Amazon | Target | Amazon | Target
Toys |  truck, car, dinosaur, construction, carrier, target, race, vehicles, friction, arrow |  blaster, truck, nerf, vehicle, light, starter, kit, ipad, dart |  princess, makeup, tent, art, doll, unicorn, doodle, house, little, pink |  surprise, baby, fashion, activity, unicorn, pet, puppy, dolls, hair, doll
Games |  target, blaster, gun, dart, electronic, nerf, ball, blue, astroshot, magnetic |  connect, boy, blue, foosketball, bulls, eye, sparkle, bash, baby, predictions |  unicorn, dance, rainbow, crafts, women, music, bubble, candy, disney, princess |  princess, meme, disney, minnie, story, perfection, upwords, guess, magic,
Sports |  toss, target, tee, electronic, challenge, boy magnetic, boxing, gloves, rocket |  boys, nerf, red, football, playball, athletic, bat, ball, archery, game, |  bra, big, women, beach, cotton, gym, yoga, tennis, crop, headband |  girls, bra, motion, purple, magic, slip, dye, cutout, cloud, strappy,
Books |  awesome, big, boys, house, goodnight, tree, game, growth, national, knight |  boy, big, hardy, bad, game, captain, underpants, battle, bionic, booger, |  girl, unicorn, disney, creative, movement, story, young, confident, beautiful, mystery |  girl, stories, minute, power, brave, target, baby, frozen, story, disney
Learning Material |  card, gamenote, classroom, leapfrog, friends, frustration, green, control, scholastic, teaching |  leapfrog, vtech, bilingual, words, book, real, sex, boys, gender, black |  pieces, homeschool, visual, tactile, auditory, mosaic, magnetic, fishing, wonderful, pattern |  activity, disney, princess, beads, kit, crinkle, paper, cloud, island, toy
All Items |  dinosaur, red, fire, blue, car, ball, constructions, vehicle, truck, race |  boy, war, toy, car, black, blue, man, action, star, mega |  princess, pink, unicorn, purple, doll, disney, house, perfect, rainbow, white |  princess, girl, disney, frozen, mermaid, minnie, surprise, pink, fashion, doll

###  4.2. Search Results

After query suggestions, we wanted to explore the role of _search results_ in
manifesting gender stereotypes in children’s products. With that goal, we
answered two research questions.

Like before, we used items from Table 1 to formulate queries, with the
addition of expanding each query to explicitly mention gender. For example,
for “action figures”, we searched for “action figures for boys” and “action
figures for girls” in addition to the gender-neutral “action figures for
kids”. We analyzed every search result for each item and used the title of
retrieved results for content analysis.

(a) Amazon search results

(b) Target search results

Figure 3. Gender associations for items in search results, colored by
previously-documented stereotypes in our toy source lists. Yellow represents
previously-documented girls’ items, green are the previously-documented boys’
items, and rest are neutral items.

RQ2a. How do search results show stereotypical responses to the gender
mentioned in the query?

For this analysis, we issued searches with our list of toys as well as generic
searches (toys, books, sports, games, and learning materials). To detect if
there is any difference in retrieved results with the change of gender in the
query for the same item or generic search, we examined the terms frequently
appearing in search results for only the gender specific queries.

We collected the titles of retrieved products and used NLTK (Loper and Bird,
2002) to filter the text, tokenized it (RegexpTokenizer), removed stopwords
(NLTK’s English corpus), and lemmatized the resulting terms
(WordNetLemmatizer). We then identified nouns and adjectives using pos_tag
from NLTK from the processed words. We counted the number of occurrence of
each token in search results and identify unique keywords that only appeared
with specific gendered search for an item (e.g. in “toys for girls” but not in
“toys for boys”). Table 2 shows the list of most-frequent terms associated
with individual gendered search for each general query and for the set of all
item queries, ordered by decreasing frequency.

Previous studies on gender studies associated with children’s products
identified common traits in gender stereotyped products (Murnen et al., 2016;
Cherney, 2018; Coyne et al., 2016; Azmi et al., 2021). For example, boy-
targeted items often represent strong, action, assertiveness, violence,
dominance, and aggressiveness. One the other hand, girl-targeted items often
demonstrated domestic characteristics, soft or kind nature, and appearance
focused (Blakemore and Centers, 2005). Previous studies already established
that ”pink” is targeted for girls and ”blue” for boys (Cherney, 2018). We have
identified the persistence of the trend of associating stereotypes with gender
in search results.

  * •

In both systems, “toys for boys” generates words that are typically
represented only as masculine products i.e. “truck”, “car”, “airplane”,
“arrow”, etc. and “toys for girls” generates feminine products like,
“princess”, “makeup”, “doll”, etc. The keywords from retrieved results show
clear distinctions between genders which reflect the gender-stereotyped
categorization of products in previous studies.

  * •

In “games for boys”, top results included violent terms like guns, bullet,
blaster, battle, whereas frequent keywords like “princess”, “rainbow”,
“unicorn”, associated with “games for girls” show attractive characteristics
(Culhane and Bazeley, 2019; Coyne et al., 2016).

  * •

We can see further distinctions in “sports”. “sports for boys” elicits
keywords that are related to different sports and activities, whereas “sports
for girls” mostly represents appearance-based keywords like clothing
accessories, consistent with the findings of Murnen et al. (2016) that boys’
items are generally action-based whereas girls’ items are more appearance-
based.

  * •

Gendered search for “books” shows the similar trend by using adventurous terms
to represent “books for boys” and magical terms to represent “books for girls”
(Lewis et al., 2020).

  * •

When searching for specific items instead of broad categories, we still saw
gender-specific result keywords. “pink” is the most common terms for
representing items for girls and “blue” for boys. Disney princesses like Elsa,
Ariel, Belle appeared frequently in girl-targeted results whereas superheroes
like Batman and Superman frequently appeared in boy-targeted results.

  * •

In addition, there is a difference in adjectives in product representation
while targeting certain genders. “adventure”, “action”, “electronic”,
“magnetic”, “strategic”, and “crazy” are frequently found in boy-targeted
products, whereas, “beautiful”, “magical”, “mystery”, “musical”, and
“colorful” are most commonly used only for girls.

These differences in gender-targeted search results qualitatively indicates
that the selected e-commerce search systems replicated and reinforced gender
stereotypes through search results in order to target a specific gender, in
ways that sometimes correspond to specific previously-documented stereotype
associations.

RQ2.b Do e-commerce search systems associate items with genders through search
results?

This analysis uses all three of the queries for each toy (gender-neutral plus
the two gendered queries) for each item in Table 1. We then investigated the
differences in sets of items retrieved, instead of their keywords, as we
changed the gender of the query, which gender the search engine associates
more strongly with that toy category. We do this by measuring the similarity
between the gender-neutral result set and each of the gender-targeted result
sets; if the neutral search results closely match with a specific gendered
search results, we identified that product to be associated with that gender.
For example, if search results for “dolls for kids” has more items in common
with search results for “dolls for girls” than “dolls for boys”, we take that
as evidence that the system associates dolls more strongly with girls than
boys.

We measured the Jaccard similarity between the set of items retrieved by the
gender-neutral query and the sets produced by boy-specific and girl-specific
queries, resulting in Boy and Girl scores for each query on each systems.

Table 3. Top-20 gender specific items assumed by observed systems along with their similarity score with gender-neutral search results. Items are sorted based on difference between similarity Score of boy-Specific and girl-Specific results.  Amazon | Target
---|---
Item | Similarity Score | Difference | Prev. Label | Item | Similarity Score | Difference | Prev. Label
Top 20 Boy-Aligned Items
| Boy | Girl | Boy-Girl |  |  | Boy | Girl | Boy-Girl |
military toys | 0.21 | 0.04 | 0.17 | Boy | power ranger valentine | 0.5 | 0.0 | 0.50 | Girl
football | 0.2 | 0.061 | 0.13 | Boy | toy drones | 0.33 | 0.0 | 0.33 | Boy
outer-space toys | 0.28 | 0.15 | 0.13 | Boy | toy robots | 0.33 | 0.0 | 0.33 | Boy
doll humanoid | 0.2 | 0.09 | 0.10 | Boy | monster trucks | 0.33 | 0.0 | 0.33 | Boy
hockey stick | 0.15 | 0.05 | 0.10 | Boy | nerf guns | 0.33 | 0.0 | 0.33 | Boy
spiderman | 0.15 | 0.05 | 0.10 | Boy | soldier toys | 0.31 | 0.0 | 0.31 | Boy
wwe action figure | 0.176 | 0.08 | 0.09 | Boy | dinosaur toys | 0.5 | 0.2 | 0.30 | Boy
superhero costume | 0.17 | 0.07 | 0.09 | Boy | ninja costume | 0.27 | 0.02 | 0.24 | Boy
spiderman costume | 0.19 | 0.10 | 0.09 | Boy | pokemon cards | 0.42 | 0.24 | 0.20 | Boy
sword toy | 0.23 | 0.14 | 0.09 | Boy | baking kit | 0.16 | 0.0 | 0.16 | Girl
construction toys | 0.18 | 0.09 | 0.09 | Boy | princess sword | 0.33 | 0.16 | 0.16 | Girl
dinosaur toy | 0.20 | 0.12 | 0.08 | Boy | airport toys | 0.13 | 0.0 | 0.16 | Boy
batman costume | 0.19 | 0.11 | 0.08 | Boy | construction toys | 0.19 | 0.07 | 0.16 | Boy
space station toys | 0.31 | 0.23 | 0.08 | Boy | cupcake maker | 0.13 | 0.0 | 0.13 | Girl
weather forecasting toys | 0.29 | 0.21 | 0.08 | Boy | remote controlled car | 0.3 | 0.17 | 0.13 | Boy
miniature guns | 0.33 | 0.26 | 0.07 | Boy | gi joe action figure | 0.17 | 0.05 | 0.12 | Boy
action figures | 0.13 | 0.05 | 0.07 | Boy | winnie the pooh | 0.11 | 0.0 | 0.11 | Neutral
toy animals | 0.16 | 0.1 | 0.07 | Boy | playdoh | 0.29 | 0.18 | 0.10 | Neutral
ninja costume | 0.19 | 0.12 | 0.07 | Boy | racetrack | 0.12 | 0.02 | 0.10 | Boy
grill | 0.19 | 0.13 | 0.06 | Boy | wagon | 0.09 | 0.0 | 0.09 | Boy
Top 20 Girl-Aligned Items
vanity set | 0.08 | 0.24 | -0.15 | Girl | fashion doll | 0.0 | 0.5 | -0.50 | Girl
play makeup | 0.14 | 0.26 | -0.11 | Girl | disney doll | 0.0 | 0.5 | -0.50 | Girl
pink doll | 0.05 | 0.17 | -0.11 | Girl | toy helicopter | 0.02 | 0.41 | -0.40 | Boy
beads | 0.05 | 0.17 | -0.11 | Girl | tiara | 0.0 | 0.4 | -0.40 | Girl
dollhouse | 0.09 | 0.2 | -0.11 | Girl | domestic toys | 0.0 | 0.33 | -0.33 | Girl
dorothy costume | 0.12 | 0.23 | -0.10 | Girl | tea party set | 0.16 | 0.5 | -0.33 | Girl
car toys | 0.06 | 0.16 | -0.10 | Boy | princess costume | 0.0 | 0.32 | -0.31 | Girl
makeup kit | 0.14 | 0.25 | -0.10 | Girl | sully costume | 0.02 | 0.32 | -0.29 | Girl
unicorn | 0.04 | 0.14 | -0.09 | Girl | beads | 0.2 | 0.31 | -0.28 | Girl
drum set | 0.06 | 0.15 | -0.09 | Neutral | remote controlled helicopter | 0.06 | 0.33 | -0.27 | Boy
castle tent | 0.11 | 0.2 | -0.09 | Girl | barbie costume | 0.02 | 0.29 | -0.27 | Girl
baby doll stroller | 0.11 | 0.21 | -0.092 | Girl | spongebob squarepants | 0.0 | 0.25 | -0.25 | Neutral
painting kit | 0.05 | 0.14 | -0.08 | Girl | ken doll | 0.08 | 0.33 | -0.25 | Girl
jewelry | 0.01 | 0.1 | -0.08 | Girl | my little pony | 0.0 | 0.23 | -0.23 | Girl
baby doll | 0.08 | 0.15 | -0.07 | Girl | wwe action figure | 0.043 | 0.26 | -0.22 | Boy
my little pony | 0.1 | 0.21 | -0.07 | Girl | lego cars | 0.0 | 0.2 | -0.20 | Boy
sully costume | 0.14 | 0.21 | -0.07 | Girl | craft toys | 0.0 | 0.2 | -0.20 | Girl
tea party set | 0.19 | 0.27 | -0.07 | Girl | vanity set | 0.07 | 0.26 | -0.18 | Girl
tea set | 0.15 | 0.22 | -0.07 | Girl | baby doll stroller | 0.15 | 0.32 | -0.16 | Girl
doll | 0.01 | 0.07 | -0.069 | Girl | toy animals | 0.0 | 0.16 | -0.16 | Girl

Table 3 presents the top 20 individual boy- and girl-aligned items, based on
the difference in boy and girl similarity scores. We can see that item lists
vary between systems. Figure 3(a) and 3(b) show the derived gender affinity
match with the previously-documented gendered item list. Both systems
displayed a tendency of identifying items as gender-specific reflecting the
previously-documented gender stereotypes to a significant extent: 97% of the
top gender-aligned items from Amazon match with the previously-documented
gender label of these items, while 72% of the top Target items do. This
indicates the potential of reinforcing and preserving gender stereotypes in
children’s products through search results.

Key findings: In RQ2, we explored the existence of gender stereotypes in
search results for children’s products. We made the following observations:

  * •

For the same item, retrieved results change with the presence of gender in the
query.

  * •

Products are often associated with gender stereotyped keywords. IR systems
reflect and reinforce that pattern by manifesting stereotypical gender-typed
contents through retrieved results targeting a specific gender.

  * •

Both systems show a similar tendency in replicating and displaying gender
stereotypes through search results.

##  5\. Discussion

Through analyzing query suggestions, we observed that production e-commerce
systems often associates gender with items (by suggesting gender-related
queries when searching for the item), and that these associations reflect the
common social practice of categorizing children’s products by gender. Although
the degree of this tendency of gender association varies across different
systems, no e-commerce system in our experiment showed complete exception from
this behavior. By suggesting target gender with children’s items, these
systems reproduce previously-documented gender stereotypes users’ search
experiences and may influence them to select accordingly.

Content analysis of search results allowed us to explore if and how e-commerce
search systems manifest gender stereotypes in children’s items through search
results. We observed that content representation changes with presence of
gender in the query, supporting pre-defined gender-specific item
characteristics. This finding implies that e-commerce search systems had the
tendency of categorizing items based on gender and promoting stereotypical
gendered target of children’s products.

The similarity between gender specific search and gender neutral search for
same item let us identify if e-commerce search systems have the potential to
manifest gender stereotypes by assuming item appropriateness for a specific
gender. Our results from analyzing two of the most popular e-commerce sites
show that e-commerce search systems can infer gender affiliation with items
and retrieve relevant information accordingly which indicates the possibility
of these systems reinforcing the practice of gender-targeted children’s
products marketing.

Our findings provide evidence that e-commerce search engines can reproduce
gender stereotypes through query suggestions and search results, and we
present initial methods for quantifying this effect. This demonstrates the
importance of studying e-commerce platforms, both in their explicit design and
emerging properties of their supporting systems. Even when explicit gender
categorizations are removed from physical stores and e-commerce site layouts,
as Target has done, a site’s search engine may still carry latent
categorization that can lead the system to replicate gender stereotypes.

##  6\. Limitations and Future Directions

In this paper, we have provided initial evidence that search systems can
propagate stereotypical gender associations into search results, particularly
for children’s products.

As an early exploratory study, there are a number of limitations we
acknowledge in this work. There is much more work to do to fully and more
precisely understand this phenomenon, and we hope our study provides a useful
foundation and motivation for future work that addresses many of these
limitations.

Our toy sets lists are based on lists compiled by previous research studies
and advocacy campaigns to identify toys with gender stereotyped associations;
this collection is not necessarily representative and is limited in size. New
data collection strategies, possibly based on surveys, to identify a wider
range of products and their stereotypical gender associations will enable a
more thorough and representative study of the phenomenon.

We then directly used the items as search queries, because logs of actual
queries issued by users of an e-commerce system are often not publicly
available, particularly when children might be searching. Collecting and
analyzing actual queries, particularly queries written by children, would help
us better measure system responses to their actual use. User studies with
children, parents, and educators would provide understanding of their
perception and preference about gendered-typed toys and the concerns of early
childhood exposure of gender stereotypes through IR systems.

Our content analysis focuses on retrieved product titles and extracted
keywords; more in-depth analysis that uses more content (product images,
descriptions, etc.) and more sophisticated analysis techniques are welcome.
Also, by directly measuring the similarity in retrieved products between
gender-specific and gender-neutral, we identified system tendency to associate
items with gender through search results. This has enabled us to gather
evidence of the extent of stereotype reproduction, but more work is needed to
refine and validate measurement methodologies. In the future, we can work
towards more fine grained and conclusive strategies to identify gender
stereotypes in search results.

Our main focus for this work was on e-commerce search systems considering two
vital components of these systems: query suggestions and search results for
our analysis. However, the role of recommendations in propagating gender
stereotypes associated with children’s products remains open to explore in the
future.

Finally, our search result findings are limited to two popular e-commerce
sites; other systems may well represent stereotypes in different ways (or not
at all).

Additional directions of future research regarding this issue include more
refined methods to measure gender stereotypes in IR systems, identifying and
understating the causes of these stereotypes, studying their impact, and
developing mitigation strategies.

Many researchers and advocates are concerned about gender stereotypes in
children’s early experiences, particularly through the toys they play with. We
hope this work will prove a useful foundation for researchers to join us in
understanding the role IR systems play and how they may affect children’s
development.

###### Acknowledgements.

This material is based upon work supported by the National Science Foundation
under Grant No. IIS 17-51278. We also thank our colleagues in the People and
Information Research Team (PIReT) for their ongoing support, and particularly
Dr. Sole Pera for her valuable feedback and guidance on this project.

## References

  * (1)
  * Abbasi et al. (2019) Mohsen Abbasi, Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2019\.  Fairness in representation: quantifying stereotyping as a representational harm. In _Proceedings of the 2019 SIAM International Conference on Data Mining_. SIAM, 801–809.
  * Ahn et al. (2022) Jungyong Ahn, Jungwon Kim, and Yongjun Sung. 2022.  The effect of gender stereotypes on artificial intelligence recommendations.  _Journal of Business Research_ 141 (2022), 50–59.
  * Alghowinem (2018) Sharifa Alghowinem. 2018\.  A safer youtube kids: An extra layer of content filtering using automated multimodal analysis. In _Proceedings of SAI Intelligent Systems Conference_. Springer, 294–308.
  * Auster and Mansbach (2012) Carol J Auster and Claire S Mansbach. 2012.  The gender marketing of toys: An analysis of color and type of toy on the Disney store website.  _Sex roles_ 67, 7 (2012), 375–388.
  * Azmi et al. (2021) Nor Jijidiana Azmi, Isyaku Hassan, Radzuwan Ab Rashid, Zulkarnian Ahmad, Nor Azira Aziz, and Qaribu Yahaya Nasidi. 2021.  Gender Stereotype in toy advertisements on social networking sites.  _Online Journal of Communication and Media Technologies_ (2021).
  * Baker and Potts (2013) Paul Baker and Amanda Potts. 2013.  ‘Why do white people have thin lips?’Google and the perpetuation of stereotypes via auto-complete search forms.  _Critical discourse studies_ 10, 2 (2013), 187–204.
  * BEAM (2021) ADAM BEAM. 2021\.  California law requires gender-neutral area in some stores.  _AP NEWS_ (Oct 2021).  https://apnews.com/article/business-gavin-newsom-california-state-legislature-legislature-6ee331cbf5eb7a22c046f5ed528b42f9
  * Berenbaum (2018) Sheri A Berenbaum. 2018\.  Beyond pink and blue: The complexity of early androgen effects on gender development.  _Child development perspectives_ 12, 1 (2018), 58–64.
  * Blakemore and Centers (2005) Judith E Owen Blakemore and Renee E Centers. 2005.  Characteristics of boys’ and girls’ toys.  _Sex roles_ 53, 9 (2005), 619–633.
  * Cauteruccio et al. (2020) Francesco Cauteruccio, Enrico Corradini, Giorgio Terracina, Domenico Ursino, and Luca Virgili. 2020\.  Investigating Reddit to detect subreddit and author stereotypes and to evaluate author assortativity.  _Journal of Information Science_ (2020), 0165551520979869.
  * Cherney (2018) Isabelle D Cherney. 2018\.  Characteristics of masculine and feminine toys and gender-differentiated play.  (2018).
  * Coyne et al. (2016) Sarah M Coyne, Jennifer Ruh Linder, Eric E Rasmussen, David A Nelson, and Victoria Birkbeck. 2016\.  Pretty as a princess: Longitudinal effects of engagement with Disney princesses on gender stereotypes, body esteem, and prosocial behavior in children.  _Child development_ 87, 6 (2016), 1909–1925.
  * Culhane and Bazeley (2019) L Culhane and A Bazeley. 2019.  UNLIMITED POTENTIAL-Report of the commission on gender stereotypes in early childhood.  (2019).  https://www.fawcettsociety.org.uk/unlimited-potential-the-final-report-of-the-commission-on-gender-stereotypes-in-early-childhood
  * Dinella and Weisgram (2018) Lisa M Dinella and Erica S Weisgram. 2018.  Gender-typing of children’s toys: Causes, consequences, and correlates.  _Sex Roles_ 79, 5 (2018), 253–259.
  * Dragovic et al. (2016) Nevena Dragovic, Ion Madrazo Azpiazu, and Maria Soledad Pera. 2016\.  ” Is Sven Seven?” A Search Intent Module for Children. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_. 885–888.
  * Eisen et al. (2021) Sierra Eisen, Shoronda Erica Matthews, and Jamie Jirout. 2021.  Parents’ and children’s gendered beliefs about toys and screen media.  _Journal of Applied Developmental Psychology_ 74 (2021), 101276.
  * Ekstrand et al. (2019) Michael D Ekstrand, Robin Burke, and Fernando Diaz. 2019\.  Fairness and discrimination in retrieval and recommendation. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1403–1404.
  * Ekstrand et al. (2021) Michael D Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. 2021.  Fairness and discrimination in information access systems.  _arXiv preprint arXiv:2105.05779_ (2021).
  * Ekstrand and Kluver (2021) Michael D Ekstrand and Daniel Kluver. 2021.  Exploring author gender in book rating and recommendation.  _User modeling and user-adapted interaction_ (2021), 1–44.
  * Ellemers (2018) Naomi Ellemers. 2018\.  Gender stereotypes.  _Annual review of psychology_ 69 (2018), 275–298.
  * Epps-Darling et al. (2020) Avriel Epps-Darling, Romain Takeo Bouyer, and Henriette Cramer. 2020.  Artist gender representation in music streaming. In _Proceedings of the 21st International Society for Music Information Retrieval Conference (Montréal, Canada)(ISMIR 2020). ISMIR_. 248–254.
  * Fabris et al. (2020) Alessandro Fabris, Alberto Purpura, Gianmaria Silvello, and Gian Antonio Susto. 2020.  Gender stereotype reinforcement: Measuring the gender bias conveyed by ranking algorithms.  _Information Processing & Management_ 57, 6 (2020), 102377\.
  * Freeman (2007) Nancy K Freeman. 2007\.  Preschoolers’ perceptions of gender appropriate toys and their parents’ beliefs about genderized behaviors: Miscommunication, mixed messages, or hidden truths?  _Early childhood education journal_ 34, 5 (2007), 357–366.
  * Fulcher and Coyle (2018) Megan Fulcher and Emily F Coyle. 2018.  Working at play: Gender-typed play and children’s visions of future work and family roles.  (2018).
  * Fulcher and Hayes (2018) Megan Fulcher and Amy Roberson Hayes. 2018.  Building a pink dinosaur: The effects of gendered construction toys on girls’ and boys’ play.  _Sex Roles_ 79, 5 (2018), 273–284.
  * Gao and Shah (2021) Ruoyuan Gao and Chirag Shah. 2021.  Addressing bias and fairness in search systems. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2643–2646.
  * Kate (2017) Crawford Kate. 2017\.  The trouble with bias.  (2017).  https://www.youtube.com/watch?v=fMym_BKWQzk
  * Kay et al. (2015) Matthew Kay, Cynthia Matuszek, and Sean A Munson. 2015\.  Unequal representation and gender stereotypes in image search results for occupations. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_. 3819–3828.
  * Kneeskern and Reeder (2020) Ellen E Kneeskern and Patricia A Reeder. 2020.  Examining the impact of fiction literature on children’s gender stereotypes.  _Current Psychology_ (2020), 1–14.
  * Kollmayer et al. (2018) Marlene Kollmayer, Marie-Therese Schultes, Barbara Schober, Tanja Hodosi, and Christiane Spiel. 2018\.  Parents’ judgments about the desirability of toys for their children: Associations with gender role attitudes, gender-typing of toys, and demographics.  _Sex Roles_ 79, 5 (2018), 329–341.
  * Leanne (2015) Etheridge Leanne. 2015\.  Who gets to play? What do toy ads on UK TV tell children about boys’ and girls’ play?  _Let Toys Be Toys_ (2015).  https://www.lettoysbetoys.org.uk/research/tv-advertising-research/
  * Lewis et al. (2020) Molly Lewis, M Cooper Borkenhagen, Ellen Converse, Gary Lupyan, and Mark S Seidenberg. 2020\.  What might books be teaching young children about gender?  (2020).
  * Liben et al. (2018) Lynn S Liben, Kingsley M Schroeder, Giulia A Borriello, and Erica S Weisgram. 2018.  Cognitive consequences of gendered toy play.  (2018).
  * Loper and Bird (2002) Edward Loper and Steven Bird. 2002.  Nltk: The natural language toolkit.  _arXiv preprint cs/0205028_ (2002).
  * Madrazo Azpiazu et al. (2018) Ion Madrazo Azpiazu, Nevena Dragovic, Oghenemaro Anuyah, and Maria Soledad Pera. 2018.  Looking for the movie Seven or Sven from the movie Frozen? A Multi-perspective strategy for recommending queries for children. In _Proceedings of the 2018 Conference on Human Information Interaction & Retrieval_. 92–101.
  * Magno et al. (2016) Gabriel Magno, Camila Souza Araújo, Wagner Meira Jr, and Virgilio Almeida. 2016.  Stereotypes in search engine results: understanding the role of local and global factors.  _arXiv preprint arXiv:1609.05413_ (2016).
  * Makhortykh et al. (2021) Mykola Makhortykh, Aleksandra Urman, and Roberto Ulloa. 2021\.  Detecting race and gender bias in visual representation of AI on web search engines. In _International Workshop on Algorithmic Bias in Search and Recommendation_. Springer, 36–50.
  * Mikolov et al. (2013) Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013\.  Linguistic regularities in continuous space word representations. In _Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies_. 746–751.
  * Milano et al. (2020) Silvia Milano, Mariarosaria Taddeo, and Luciano Floridi. 2020.  Recommender systems and their ethical challenges.  _AI & SOCIETY_ 35, 4 (2020), 957–967.
  * Murnen (2018) Sarah K Murnen. 2018\.  Fashion or action? Gender-stereotyped toys and social behavior.  (2018).
  * Murnen et al. (2016) Sarah K Murnen, Claire Greenfield, Abigail Younger, and Hope Boyd. 2016.  Boys act and girls appear: A content analysis of gender stereotypes associated with characters in children’s popular culture.  _Sex roles_ 74, 1 (2016), 78–91.
  * Noble (2018) Safiya Umoja Noble. 2018\.  _Algorithms of oppression_.  New York University Press.
  * Otterbacher (2018) Jahna Otterbacher. 2018\.  Addressing social bias in information retrieval. In _International Conference of the Cross-Language Evaluation Forum for European Languages_. Springer, 121–127.
  * Raj et al. (2021) Amifa Raj, Ashlee Milton, and Michael D. Ekstrand. 2021\.  Pink for princesses, blue for superheroes: The need to examine gender stereotypes in kid’s products in search and recommendations.  _arXiv preprint arXiv:2105.09296_ (2021).
  * Raj et al. (2020) Amifa Raj, Connor Wood, Ananda Montoly, and Michael D. Ekstrand. 2020.  Comparing fair ranking metrics.  _arXiv preprint arXiv:2009.01311_ (2020).
  * Richardson and Simpson (1982) John G Richardson and Carl H Simpson. 1982.  Children, gender, and social structure: An analysis of the contents of letters to Santa Claus.  _Child Development_ (1982), 429–436.
  * Roy and Ayalon (2020) Senjooti Roy and Liat Ayalon. 2020.  Age and gender stereotypes reflected in Google’s “autocomplete” function: The portrayal and possible spread of societal stereotypes.  _The Gerontologist_ 60, 6 (2020), 1020–1028.
  * Russell (2021) Helen Russell. 2021\.  Lego to remove gender bias from its toys after findings of child survey.  _The Guardian_ (Oct 2021).  https://www.theguardian.com/lifeandstyle/2021/oct/11/lego-to-remove-gender-bias-after-survey-shows-impact-on-children-stereotypes
  * Schütze et al. (2008) Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. 2008.  _Introduction to information retrieval_. Vol. 39.  Cambridge University Press Cambridge.
  * Seitz et al. (2020) Maximilian Seitz, Jan Lenhart, and Nina Rübsam. 2020\.  The effects of gendered information in stories on preschool children’s development of gender stereotypes.  _British Journal of Developmental Psychology_ 38, 3 (2020), 363–390.
  * Skočajić et al. (2020) Milica M Skočajić, Jovan G Radosavljević, Milica G Okičić, Ivana O Janković, and Iris L Žeželj. 2020\.  Boys just don’t! Gender stereotyping and sanctioning of counter-stereotypical behavior in preschoolers.  _Sex Roles_ 82, 3 (2020), 163–172.
  * Souza et al. (2021) Rosilene Rodrigues de Souza, Jéssica Renata Bastos Depianti, Renata de Moura Bubadué, Camille Xavier de Mattos, Julia Maricela Torres Esperón, and Ivone Evangelista Cabral. 2021.  For boys or girls? Gender cultural representations in toy packaging.  _Texto & Contexto-Enfermagem_ 30 (2021).
  * Spinner et al. (2018) Lauren Spinner, Lindsey Cameron, and Rachel Calogero. 2018\.  Peer toy play as a gateway to children’s gender flexibility: The effect of (counter) stereotypic portrayals of peers in children’s magazines.  _Sex roles_ 79, 5 (2018), 314–328.
  * Stephanie (2021) Chevalier Stephanie. 2021\.  Market share of leading retail e-commerce companies in the United States.  _Statitia_ (2021).  https://www.statista.com/statistics/274255/market-share-of-the-leading-retailers-in-us-e-commerce/
  * Sweeney (2013) Latanya Sweeney. 2013\.  Discrimination in online ad delivery.  _Commun. ACM_ 56, 5 (2013), 44–54.
  * Urman et al. (2021) Aleksandra Urman, Mykola Makhortykh, Roberto Ulloa, and Juhi Kulshrestha. 2021.  Where the Earth is flat and 9/11 is an inside job: A comparative algorithm audit of conspiratorial information in web search results.  _arXiv preprint arXiv:2112.01278_ (2021).
  * Voorhees et al. (2004) Ellen M Voorhees et al. 2004\.  Overview of TREC 2004.. In _Trec_.
  * Weisgram and Dinella (2018) Erica S Weisgram and Lisa M Dinella. 2018.  _Gender typing of children’s toys: How early play experiences impact development_.  JSTOR.
  * Weisgram et al. (2014) Erica S Weisgram, Megan Fulcher, and Lisa M Dinella. 2014\.  Pink gives girls permission: Exploring the roles of explicit gender labels and gender-typed colors on preschool children’s toy preferences.  _Journal of Applied Developmental Psychology_ 35, 5 (2014), 401–409.