# Imagine a dragon made of seaweed: How images enhance learning in Wikipedia

Anita Silva♠, Maria Tracy♠, Katharina Reinecke♠, Eytan Adar♢, Miriam Redi♣
$\spadesuit$ Paul G. Allen School of Computer Science & Engineering,
University of Washington; $\diamondsuit$ School of Information, University of
Michigan; $\clubsuit$Wikimedia Foundation

###### Abstract.

Though images are ubiquitous across Wikipedia, it is not obvious that the
image choices optimally support learning. When well selected, images can
enhance learning by dual coding, complementing, or supporting articles. When
chosen poorly, images can mislead, distract, and confuse. We developed a large
dataset containing 470 questions & answers to 94 Wikipedia articles with
images on a wide range of topics. Through an online experiment (n=704), we
determined whether the images displayed alongside the text of the article are
effective in helping readers understand and learn. For certain tasks, such as
learning to identify targets visually (e.g., which of these pictures is a
“gujia”?)111For the curious: Gujia are sweet moon-shaped dumplings native to
the Indian subcontinent. , article images significantly improve accuracy.
Images did not significantly improve general knowledge questions (e.g., where
are gujia from?). Most interestingly, only some images helped with visual
knowledge questions (e.g., what shape is a gujia?). Using our findings, we
reflect on the implications for editors and tools to support image selection.

Wikipedia, learning, images, question and answer dataset

††conference: ; ; ††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Human-
centered computing Wikis††ccs: Applied computing Interactive learning
environments

##  1\. Introduction

One of the main motivations for visiting Wikipedia is intrinsic learning
(Singer et al., 2017). This is due in large part to the significant editorial
attention that curates Wikipedia content toward a clear, factual knowledge
repository. While most attention is focused on textual content, Wikipedia
sites are a multimedia experience. Images complement and augment millions of
pages. For example, the reader visiting the Leafy Seadragon page may find it
difficult to visualize how, “the lobes of skin that grow on the leafy
seadragon provide camouflage, giving it the appearance of seaweed.” (English
Wikipedia, 2022). When a Wikipedia editor introduces a high-quality image (see
Figure 1), that image can significantly illuminate the subject.

To support the use of such images, Wikimedia allows for freely licensed
multimedia files to be uploaded to the Wikimedia Commons site. As of February
2024, more than 103 million documents—largely photos, maps, and diagrams—have
been uploaded with the central purpose of providing educational benefits
(i.e., “providing knowledge; instructional or informative.” (Wikimedia
Foundation, Inc., 2022b)). In the context of Wikipedia, images can be curated,
added, removed, modified, placed into galleries, captioned, referenced, and
otherwise editorialized.

Although there is significant research on what makes for good text on
Wikipedia and how editors craft it (e.g., (Adler et al., 2008; Kittur and
Kraut, 2008; Wilkinson and Huberman, 2007)), there is, unfortunately, less
data on image use. This gap is important as good images can enhance learning
(e.g., through dual coding (Paivio, 1990)), while ‘bad’ images can distract
and confuse (e.g., through seductive details (Sundararajan and Adesope, 2020)
or structure interference (Schnotz and Bannert, 2003)). With a for-profit
textbook or encyclopedia, authors or editors may be able to find the ideal
image. In the case of Wikipedia, it is not clear that the “best”
representation of a concept can as easily be acquired or created. Wikipedia
presents an added complexity, as images and associated text are manually
curated by a distributed community of editors and must generally comply with
free-licensing rules. While the Wikimedia Commons image collection is vast, it
is still bounded by the availability of free content and by the scale of
volunteer efforts. The consequence is that we do not know if and when
Wikipedia-embedded images can help in learning.

Images can serve in multiple roles for multiple kinds of learning (see (Marsh
and White, 2003) for a survey). Critically, images on Wikipedia are not
isolated—they are embedded in the context of text. Thus, an image can function
in a way that closely relates to that text (e.g., to exemplify a concept) but
may also have little relation to it (e.g., a decorative purpose) or go beyond
it (e.g., interpret). In the case of Wikipedia, decoration roles are unlikely,
as there is a strong emphasis on the use of images for educational purposes
(Wikimedia Foundation, Inc., 2022b). Similarly, interpretive roles should be
avoided as they may go against Wikipedia’s ‘No original research’ policy
(Wikimedia Foundation, Inc., 2022g). Although this leaves most Wikipedia
images squarely in the ‘close relation to text’ category, these images can
still support different use cases. Maps can help the reader orient themselves
or provide context (e.g., where an unfamiliar town is situated). A flow
diagram can illustrate complex mechanisms (e.g., how the Kreb’s cycle works).

(a)

(b)

Figure 1. (a) An image of a leafy seadragon (source:
https://commons.wikimedia.org/wiki/File:Leafy_Seadragon_on_Kangaroo_Island.jpg)
used at the top of the page (within the infobox) for the English Wikipedia
page on the topic (b). This will be the first image readers encounter on the
page.

To investigate the role of images in learning on Wikipedia, this paper
investigates the following research question: Do Wikipedia articles with
images help answer knowledge comprehension questions accurately compared to
articles without an image? We address this question by designing a dataset and
a crowdsourced experiment to (i) quantify the importance of images for
knowledge acquisition on Wikipedia, and (ii) understand which image
characteristics make them useful for learning. We focus on images that are
largely illustrative of the Wikipedia concept (in the classification of (Marsh
and White, 2003) the image ‘reiterates’, or more specifically, ‘concretizes’
or ‘exemplifies’). For example, the image associated with the Leafy Seadragon
article in Figure 1 provides an illustrative example of what a leafy seadragon
looks like. A photograph of Donatello’s statue of David makes a concrete
illustration of what the statue looks like. We generated a learning assessment
dataset from 94 selected articles and corresponding images of this type. In a
pre-registered222https://aspredicted.org/JJQ_GVQ online experiment with 704
participants recruited through LabintheWild (LabintheWild, 2021) and Prolific,
we determined the helpfulness of images integrated into these articles for a
number of learning tasks. Specifically, we assessed: (1) Image Recognition
Questions–can the reader learn to distinguish a new image of a target concept
(e.g., is this a leafy seadragon)?; (2) Visual Knowledge Questions–can the
reader learn visual properties of the concept (e.g., what color are leafy
seadragons)?; and (3) General Knowledge Questions–can the reader learn general
knowledge information about the concept (e.g., where are leafy seadragons
found)? In all, we crafted 470 questions across all three types. These, of
course, assess only a small portion of possible learning objectives. From the
perspective of learning, we target the lowest levels of Bloom’s learning
objective taxonomy–Recall or Recognize (Anderson et al., 2001; Bloom et al.,
1956). Nonetheless, these objectives are consistent with those of the majority
of Wikipedia visitors in getting an overview or fact-checking (Lemmerich et
al., 2019; Singer et al., 2017).

Our results show that images can support learning, but only for some tasks. In
our study, images significantly improved Image Recognition tasks but did not
improve General Knowledge learning. However, we find that images do not appear
to hurt the learning tasks—a concern if readers overly focus on the image and
do not engage with the content. The results for answering Visual Knowledge
questions are more mixed. We found that in some cases images can be consistent
or inconsistent with textual content. For example, the text may indicate that
a specific gem’s coloring is “blue or green.” However, the image may only show
a blue example. When asked, “can the gem be green?” the reader may not be able
to correctly answer. With these inconsistent images, a reader would not answer
the question correctly if they were only focused on the image and ignored or
did not read the text. In contrast, with consistent images, the answer is
apparent from the image regardless of the text.

We identify several implications of these results, including better support
tools for Wikipedia editors, policies, and potential new forms of ‘wikiwork.’
In addition to these empirical results and design implications, we contribute
a novel dataset of articles, images, and assessments that can be used as a
baseline for evaluating images on
Wikipedia333https://github.com/datamazelab/wikiimages.

##  2\. Related Work

Literature from different disciplines informed the research in this paper. To
contextualize our work, we first discuss findings on the role of images in
learning and reading comprehension. Additionally, we place our research within
related work on Wikipedia as a learning platform and the impact of various
design choices made in the presentation of content on Wikipedia.

###  2.1. The role of text illustrations

Several projects from experimental psychology have focused on the role of
images in learning and reading comprehension. Research work investigating
images’ cognitive function (as defined by Levie and Lentz (Levie and Lentz,
1982)) to facilitate knowledge acquisition found mixed results. In some cases,
images in association with text help support learning (Guo et al., 2020),
especially when images are correctly captioned (Bernard, 1990), and when
graphics are designed to support the interpretation of textual information
(Renkl and Scheiter, 2017). On the other hand, the added visual complexity can
challenge the reader’s comprehension (Guo et al., 2020). In online settings,
images have been found to slow reading time without having a significant
influence on content retention (Beymer et al., 2007; Li et al., 2019).

Visual content in conjunction with text can have an attentional role, drawing
attention to information in textual form, or as an affective pull that
enhances emotions (Lee-Robbins and Adar, 2023; Levie and Lentz, 1982). Images
can also have an effect on reader motivation and prejudicial behavior: for
example, the presence of images of women in science books has a positive
impact on the performance of female students (Good et al., 2010).

There are a number of theories and empirical results that support the notion
that well-crafted or selected visual images can aid in learning. For example,
dual coding theory argues that two different cognitive systems (verbal and
imagery/nonverbal) retain different types of information, each with its own
advantage (Paivio, 1990). Because these systems are connected, the theory
posits that including images, “may better support retention of the material as
it provides learners with two ways to memorize information” (Vekiri, 2002).
However, certain images may disrupt learning (e.g., seductive images, which
contain interesting but irrelevant information (Harp and Mayer, 1997)) .

Although several alternative theories seek to explain the interaction between
text and visual signals, there is consistent empirical evidence that well-
selected/crafted images can aid in learning (Clark and Lyons, 2010). Our work
seeks to broadly validate these results in the context of crowd-selected
images.

###  2.2. Wikipedia as a learning platform

One of the main reasons why readers access Wikipedia is intrinsic learning
(Singer et al., 2017), especially in developing or newly industrialized
countries (Lemmerich et al., 2019). For example, Wikipedia is one of the most
visited resources for health information (Laurent and Vickers, 2009), a
pattern that was particularly evident during the COVID-19 outbreak (Diego
Saez-Trumper, 2022). Although intrinsic learning may be more common, extrinsic
motivation (e.g., grades (Head and Eisenberg, 2010)) can also drive visits. An
analysis of the motivations of Wikipedia users based on publicly available
data reveals that readers who use English Wikipedia for intrinsic learning or
school tasks are more likely to browse articles about mathematics, biology,
and history/philosophy (Wikimedia Foundation, Inc., 2022d). Research has shown
that Wikipedia is widely used by faculty and students alike (Knight and Pryke,
2012), and its integration into instructional settings can have positive
effects (Wannemacher and Schulenburg, 2010), particularly when teaching
practices include article creation (Evenstein Sigalov and Nachmias, 2017).
Education is at the core of the Wikimedia Foundation’s mission “to empower and
engage people around the world to collect and develop educational content …and
to disseminate it effectively and globally” (Wikimedia Foundation, Inc.,
2022e). Understanding how images support both real-world use and the mission
of the underlying platform becomes a critical task.

###  2.3. Design choices on Wikipedia

Researchers have investigated various aspects of the behavior of Wikipedia
readers. These include topic preferences (Lehmann et al., 2014), engagement
with links in different positions (Dimitrov et al., 2016), how readers
perceive latency and system performance on the site (Salutari et al., 2019),
and how they engage with images (Rama et al., 2022), citations (Piccardi et
al., 2020) and external links (Piccardi et al., 2021). However, while
Wikipedia is widely used in instructional settings and applications, only a
few research works explored how Wikipedia content and layout support reading
comprehension and learning.

For example, Wikipedia has been used to find prerequisite relations for online
learning courses (Sayyadiharikandeh et al., 2019), or to test the optimal font
size for online reading comprehension (Rello et al., 2016) . However, to our
knowledge, no previous work has explicitly tested the role of images in
reading and understanding Wikipedia content.

In cases where images have been the subject of research articles, it has
primarily been to analyze the quantity and diversity of Wikipedia images.
Interdisciplinary research exposed the importance of visual content in
quantifying Wikipedia’s monetary value (Heald et al., 2015; Erickson et al.,
2018). Recent work measured the diversity of Wikipedia images across languages
(He et al., 2018) and gender gaps (Beytía et al., 2022).

Few works have investigated the role of images on Wikipedia from a user
perspective. Viegas et al. focused on understanding how and why editors use
images when contributing to Wikipedia and describing the editor communities
that curate its visual content (Viegas, 2007). Navarrete et al. (Navarrete and
Villaespesa, 2020) found that images of paintings on Wikipedia are used to
illustrate generic topics beyond art, reaching a larger pool of readers.
Recently, Rama et al. studied how readers interact with images on English
Wikipedia (Rama et al., 2022) by analyzing traffic logs from Wikimedia
Foundation servers. They found that on average, readers click on images 3 out
of 100 times they read Wikipedia pages, and that page previews with images are
less likely to be loaded in full. While this work provides an interesting
quantification of readers’ engagement with images, it does not provide any
insights as to why and how readers use visual content when browsing the
encyclopedia. In this work, we begin to address these questions by studying
the effect of images on knowledge acquisition.

##  3\. A Dataset for Text and Images

To test knowledge acquisition, we first built a dataset of multiple choice
questions, with corresponding text snippets and images, based on Wikipedia
articles. Our dataset includes three types of questions, informed by prior
work on learning and information seeking in Wikipedia (Head and Eisenberg,
2010; Singer et al., 2017; Lemmerich et al., 2019):

  * •

General Knowledge Questions to test the understanding and recall of generic
knowledge. These questions ask about either historical, metric, or other non-
visual factual aspects of the article topic. Example: What culture are the
Stone spheres of Costa Rica generally attributed to?

  * •

Visual Knowledge Questions, to measure understanding and recall of the visual
characteristics of the subject of the article as described in the article
text. These questions are often about color, material, structure, design, and
other aspects that visually describe the subject of the article. Example: What
are most of the Stone spheres of Costa Rica made of?

  * •

Image Recognition Questions, to measure our visual understanding and recall of
a topic. These questions test participants’ ability to identify the image that
best depicts an article topic. Example: Which of these images correspond to
the Stone Spheres of Costa Rica?

To create these questions, we followed a semi-automated procedure that
ultimately yielded 470 questions from 94 articles.

###  3.1. Article Selection

We started by creating a controlled subset of relevant English Wikipedia
articles. We used an automated process to identify broad topical categories of
general interest. We created our dataset from a July 2019 snapshot of the
English Wikipedia and selected articles classified as Food and Drink, Arts,
Architecture, or Biology (e.g., birds, minerals, etc.), using the language-
agnostic topic model developed by Johnson et al, (Johnson et al., 2021). Of
these, only articles with a ‘lead’ image were retained (an image in the
infobox or main section of the article(Wikipedia, 2022a)). Small icons, such
as logos or flags, were ignored.

We then selected specific articles that were less well-known, since articles
that were too familiar to the broad population would likely lead to questions
that were too easy (potentially creating a high baseline score). To identify
‘less known’ articles, we sub-selected pages that had under 6000 views per
month (9th decile) and were less than 15k characters (removing any overly
detailed articles). These two requirements ensured that the articles were less
popular from the viewers’ and editors’ perspectives. We treated this as a
proxy for articles that were less likely to be familiar to an ‘average’
reader.

When creating both visual and general knowledge questions, we wanted to ensure
that the article text provided sufficient information to answer them. For
example, the article text would need to mention that Donatello’s David is in
Italy and Detroit-style pizza is traditionally rectangular. In piloting
question generation, we found that most information for visual knowledge
questions came from specific sub-sections of the articles. Specifically, these
sections were: Description, Characteristics, Architecture, Appearance, Design,
Details, Morphology, Anatomy and physiology, Construction, Size(s), and
Physiology. Although other sections may also contain visual descriptions, we
found through inspection that these covered most of the cases. Thus, we
retained articles that had at least one of these sections. In the end, we
generated a list of 31k articles. Note that these were simply used as a seed
set for our annotators. The articles roughly conformed to our views and length
requirements and had a high likelihood of being usable for question creation.

###  3.2. Question Generation

While we originally experimented with using Mechanical Turk for this process,
we found that the questions produced were unsatisfactory. They were deemed too
easy, convoluted, having non-related distractors, or poor images for the image
recognition question type. This is consistent with the observation that good
assessments are nuanced and may be difficult to create (Haladyna, 2004).
Although the questions produced by the crowd workers were largely unusable, 20
(of 60) articles that they selected were evaluated by the research team as
appropriate and used in the experiment. Hence 20 out of the 94 articles used
in this study were produced through the Mechanical Turk pilot study.

To build up the remaining set, six members of the research team manually
picked pages from the automatically-generated list of articles. Articles that
discussed abstract or extremely specialized ideas were not selected for this
study due to their conceptual difficulty. The research team then created five
questions per article: two general knowledge questions, two visual knowledge
questions, and one image recognition question, for a total of 470 questions
across 94 articles. For each question, we created a set of multiple choice
answers, containing the correct answer, and up to three distractor answers. In
addition, we specified the titles of the sections in the articles that contain
the answers to the questions.

The question generation process was highly iterative with the research team
initially working together and discussing article selection, potential
questions, and answers. After these initial discussions, each team member
picked a set of 5-10 articles and generated a set of questions and answers.
These were subsequently reviewed by other members of the team.

##### Guidelines for Question and Answers

Following the process described above, we developed a set of guidelines to
generate a robust set of knowledge acquisition questions. Questions had to be
(1) self-explanatory:, i.e., not referring to other questions and answerable
from memory without still having access to the Wikipedia article seen before;
(2) interesting, exciting questions that could be part of a trivia game; (3)
specific, namely not containing ambiguous terms.

Additionally, we developed specific guidelines for choosing general knowledge,
visual, and image recognition questions. In the structure of a learning
objective framework such as Bloom’s (Anderson et al., 2001; Bloom et al.,
1956)) the questions assess to test recall of specific facts (e.g., “the
student will recall who commissioned Donatello’s David”). Assessments at this
level are more easily structured as multiple response questions (Haladyna,
2004). Note that while these questions test comprehension (and specifically,
recall), they were not designed around deeper learning tasks (e.g.,
summarizing, inference, etc.) or knowledge dimensions (e.g., conceptual,
procedural, etc.).

Visual knowledge questions were created only using article text and did not
rely on the article’s images. We selected visual knowledge questions in such a
way that there was plausibly an image that represented the entity itself which
could be used to answer the question (e.g., a photograph or sketch of
Donatello’s David). Of course, this need not be the image that is actually
used on the Wikipedia page. When crafted this way, visual knowledge questions
tended to ask about visually salient features (e.g., color), visually
comparable features (e.g., size in relation to similar organism), or other
physical properties that could be determined with some prior knowledge (e.g,
building material). Visual knowledge questions were also tagged with a binary
value–consistent or inconsistent–indicating if the question could be answered
by only looking at the image for the article (specifically the ‘lead’ or
’main’ image in use on Wikipedia). For example, for the article about the
Carolina Reaper pepper, the image has three examples of the pepper, but very
few cues about scale. Thus, the question “When fully ripe, about how big is a
Carolina Reaper pepper?” is marked as ‘inconsistent.’ However, the question
“What texture is the Carolina Reaper Pepper?” is answerable from the image
(and thus consistent).

General knowledge questions were factual questions that did not have to do
with visual properties of the subject and could not be answered by looking at
images. These were intended to be simple, factual details, often with one-word
answers (e.g., what part of the world or from what country is something? when
was something built or created? etc.). Visual and general knowledge questions
were structured as multiple choice with a single right answer and up to three
distractors. Distractors were generated by members of our research team.
Following suggested guidelines (Haladyna, 2004), the distractors were in the
same class as the true answer (e.g., other country names in areas of the world
geographically close to the true answer). However, distractors were not
intended to create trick questions or confusion.

To create image recognition questions, our guideline suggested finding one
picture of the entity that was not in use on the Wikipedia page and a number
of distractors (all from Wikimedia Commons). To find an appropriate target
image, we suggested a number of candidates, including: lead images used in
other language pages, images previously used on the Wikidata item page, or
images in the same category in the Wikimedia Commons as the current image. For
example, for the Turkish pastry Boyoz, all language editions utilized the same
lead image. However, the Commons category page for the image
(https://commons.wikimedia.org/wiki/Category:Boyoz) provided six alternatives.
Distractor images were found by looking at related or higher-level categories
to the entity. For example, the page for Boyoz links to
https://en.wikipedia.org/wiki/List_of_pastries. From here we see other stuffed
pastry types (e.g., the Scottish Bridie), which have a related but distinct
appearance. All generated questions–general, visual, image recognition–were
later validated for ‘baseline familiarity’ (i.e., ease), as described below.

(a) Article displayed with an image

(b) Article displayed without an image

Figure 2. Examples of articles with/without images shown in our experimental
setting.

Because articles could vary significantly in detail and length, we created
excerpted versions. The excerpt included the lead (top) section of the article
and any section that was used to generate general or visual knowledge
questions. For example, the answer to “The Père David’s deer is significant to
which cultural mythology?” was found under the “Legend and cultural
significance” subheading. This subsection was thus incorporated into the
article’s excerpt (for this specific article, the except also included the
Characteristics subsection). At most an except would be comprised of 4
subsections (one for question). However, most of the excerpts were made up of
two sections (median, mean=2.38) with a mean/median word count of 373 and 359,
respectively. Using the Flesch-Kincaid grade level metric, we found the median
readability grade level for our excerpts was 12 (M = 11.84, SD = 1.98, min =
7, max = 17). This is in line with the readability distributions observed in
the past for Wikipedia (Lucassen et al., 2012).

##### Final Dataset

In summary, a total of 94 articles were used to create the dataset. The data
includes the excerpted article and the ‘lead’ image (the header/top image seen
by visitors to Wikipedia to that article). A total of 470 questions were
produced, with five questions per article (two general knowledge questions,
two visual knowledge questions, and one image recognition question). We
provide this dataset in our supplemental materials
(https://github.com/datamazelab/wikiimages).

##  4\. Online Experiments

Our online experiment was primarily aimed at answering: [RQ1] Do Wikipedia
articles with images help participants answer questions more accurately
compared to articles without an image? and [RQ2] What characteristics of the
images are most helpful in answering the different question types?

Our baseline experiment measures participant performance without the presence
of Wikipedia article excerpts and images. For the main experiment, we designed
a within-subjects online experiment using our full knowledge comprehension
dataset. The experiments were approved by the Institutional Review Board at
the University of Washington. The main experiment was preregistered on
AsPredicted (https://aspredicted.org/JJQ_GVQ). We slightly rephrased the
preregistered research question for conciseness and added RQ2, which the
preregistration did not explicitly list (though it was implied through the
types of analyses).

###  4.1. Baseline Experiment

Our baseline experiment was designed to determine the difficulty of the
questions in our dataset without showing the content or images of the article.

The baseline experiment began with an informed consent form and a brief
overview of the study, before participants were asked 12 questions total. Six
(6) questions belonged to one article with one (1) familiarity question (prior
experience with the article’s topic), two (2) general knowledge questions, two
(2) visual knowledge questions, and a final image recognition question (1).
The familiarity (article knowledge) question included five options ranging
from “I have never heard of it” to “I am an expert on this topic.” A second
set of six questions followed the same format, but were asked on a different
article topic. Questions were asked without participants seeing the Wikipedia
article. Participant responses were collected for all questions associated
with the 94 articles in our dataset. All collected metrics are described in
Section 4.4.

Figure 3. Procedure Flow Diagram for the Baseline Study Figure 4. Procedure
Flow Diagram for our main experiment

###  4.2. Participants

##### Participants

We recruited participants from Prolific, an online research study platform. A
total of 51 Prolific participants completed the baseline experiment. Each
received $0.25/min ($15/hr) for their participation. Only 47 data entries out
of 51 were used (four participants indicated that their data should not be
included, e.g., because of self-acknowledged cheating). Of the 47, 47.82%
self-identified as female, 50% as male, and 2.17% participants answered using
other expressions (e.g., non-binary, bigender, etc.). The max age of the
participants was 69, and the minimum age recorded was 18 (M = 36.23, SD =
12.35). Two different countries of origin were reported by the participants:
United Kingdom (91.5%), and United States (8.5%). A majority (91.11%) were
English speakers, 4.44% Lithuanian, and 4.44% German speakers. The
distribution of English proficiency was 86.95% advanced, 10.86% upper-
intermediate, and 2.17% intermediate. The distribution of education levels was
21.73% high school, 26.08% college, 10.86% professional school, 34.78%
graduate school, and 6.52% Ph.D. Participants took at most 7 minutes in the
study and at least 1.5 minutes (M = 3.11, SD = 1.06).

###  4.3. Main Experiment

The main experiment began with an informed consent form and a brief overview
of the study, as seen in the upper part of Figure 4. A practice trial asked
participants to try their best and answer the randomly chosen general
knowledge, visual knowledge, and image recognition questions seen on the same
page. This allowed all participants to see the range of questions they might
be asked during the experiment. As with the baseline experiment, participants
were asked about their familiarity with a randomly chosen Wikipedia article
topic on the following page.

The main part of the experiment consisted of three trials, each including two
parts: reading excerpts from a Wikipedia article and answering three questions
for each article on the following pages. The articles were randomly selected
from our dataset and randomly shown with or without images. Before displaying
each article, the familiarity question was asked.

Participants received instructions to read the article and that they would be
asked questions about it on the next page, without having access to the
article at that point. This prevented participants from being able to refer to
the content while being assessed on their retention. The pages used to display
the excerpted article mimicked the design of standard Wikipedia articles
(e.g., Figure 2(a) and Figure 2(b)). The study display was accessible to
different device users through a feature that responded to different screen
sizes.

There was a 50% chance for a participant to see an image with their article.
If a Wikipedia article was shown with the image, we used the image from the
original Wikipedia article in its original resolution, preloading it to avoid
viewing delays that would impact our time measures. Images were displayed in a
location similar to their placement on the original Wikipedia pages (on the
top right of the article for tablet and computer users or centered on the top
for smaller device users, see Figure 2(a)).

The three comprehension questions (general knowledge, visual knowledge, and
image recognition) were displayed on three separate pages. All three were
multiple choice and included up to four answer options. Recall that when
generating the question set, we created two general and two visual knowledge
questions per article as discussed in Section 3.2. For these question types,
we randomly chose to display one of the two choices on their respective pages.

Participants received instant visual and audio feedback on their general
knowledge, visual knowledge, and image recognition question selections. In
between trials, participants were given the option to take a break while
reading a random fun fact taken from the Wikipedia ‘unusual articles page’
(English Wikipedia, 2021). In total, each participant completed twelve
questions, four for each of the three articles (familiarity/article knowledge
question, general knowledge question, visual knowledge question, and an image
recognition question).

The collected data from this experiment contains unevenly distributed answers
per question. Each participant was randomly presented with three articles (out
of 94 total articles) and three questions per article (out of 5 possible).
After each article that participants viewed, a page with a general knowledge
question and a page with a visual knowledge question were randomly displayed.
The general and visual knowledge questions were randomly selected from two
generated options for each type. Due to the random selection of questions,
even if two participants responded to questions created for the same article,
they did not necessarily answer the same visual and general knowledge
questions for that article.

Collected data for the main experiment includes question responses to 90
articles out of 94. A total of 368 question responses were collected for 470
generated questions, excluding familiarity (article knowledge) questions. The
average number of answers collected per question is 18.29 (SD$=$11.23,
max$=$74 and min$=$2).

At the end of the experiment, we asked participants to answer demographic
questions about their level of education, country of origin, age, gender,
native language, if they have done the study before, and English reading
proficiency. The next page asked if they had technical issues or cheated at
any point in the study to determine whether their data should be excluded.
Finally, they saw a results page including a personalized “Wiki Knowledge
Score” that took into account how long they took and how well they answered
each question compared to others. An extra slide was used for Prolific
participants to get their Prolific ID and to display a completion code. The
study took up to 10 minutes to complete and was entirely in English.

The experiment design deviated slightly from our preregistered design. Upon
initial analysis, we found removing the data collected for the first article’s
questions necessary to account for participants getting used to the experiment
structure. We further discuss excluding this data in the Results section and
provide the results when the first article’s questions are included in the
supplemental files.

We launched the experiment on LabintheWild (LabintheWild, 2021), a volunteer-
based experiment platform where participants receive personalized feedback in
exchange for study participation, and on Prolific, where participants receive
financial compensation. This dual-platform strategy had two functions: (1) it
allowed us to quickly recruit a sufficient number of participants to support
our statistical analysis, and (2) the two populations roughly mimic the mix of
intrinsic and extrinsic motivators of those reading Wikipedia (e.g., personal
curiosity and grades, or in the case of Prolific, money). Since LabintheWild
participants are intrinsically motivated to test themselves and perform well,
LabintheWild studies have been repeatedly shown to result in higher data
quality than studies on paid online recruitment platforms, with participants
paying more attention, providing more consistent responses and scoring higher
(Ye et al., 2017; August and Reinecke, 2019). Prolific participants were from
the US and UK and were paid 50 cents per article/question set (yielding
roughly $15/hour).

A total of 748 participants attempted the experiment, 445 through
LabintheWild, and 303 through Prolific. We excluded 37 LabintheWild
participants from the data set. These were removed if they completed the study
in less than a minute (13), or indicated reasons for excluding the data (e.g.,
self-reported cheating) (8) or technical issues (16). The latter two variables
are routinely collected in LabintheWild experiments to give participants an
option to indicate that their data should be excluded from analysis.
Collecting the same information on Prolific, we also removed 7 out of 303
Prolific participants (six for technical issues and one for finishing in under
a minute). This resulted in a total of 704 participants, 408 (57.95%) from
LabintheWild and 296 (42.04%) from Prolific.

(a) General knowledge question

(b) Visual knowledge question

(c) Image recognition question

Figure 5. Examples of (a) General knowledge questions, (b) Visual knowledge
questions and (c) Image recognition questions from our experimental setting.

Among our 704 participants, 47.57% self-identified as female, 48.71% as male,
and 3.70% participants answered using other expressions (eg. non-binary,
bigender, etc.). Participants were between 10 and 89 years old (M = 32.23, SD
= 23.60). In total, 53 different countries of origin were reported by the
participants, the top 5 being the United Kingdom (44.79%), United States
(29.10%), Canada (4.27%), Germany (3.28%), Australia (2.13%), and France
(1.42%). Participants indicated being native speakers of 31 different
languages, the majority being English (77.23%). The distribution of English
proficiency was 82.91% advanced, 12.26% upper-intermediate, 3.06%
intermediate, 1.02% beginner, and .07% basic. The distribution of education
levels was 3.43% pre-high school, 15.04% high school, 35.81% college, 9.31%
professional school, 27.07% graduate school, 7.16% PhD, and 2.14%
postdoctoral. Participants took at most 28 minutes on the study and at least 2
minutes (M = 8.59, SD = 4.22).

###  4.4. Metrics

For each participant, article, and question, we recorded several metrics for
our analysis.

####  4.4.1. Participants’ Characteristics

We stored for each participant:

  * •

Participant ID, recorded anonymously using UUID format.

  * •

English Proficiency, taken from the demographics form, can take one of the
following values: beginner, basic, intermediate, upper-intermediate, and
advanced.

  * •

Education Level, taken from the demographics form, whose values can be: pre-
high school, high school, college, graduate school, professional school,
postdoctoral, PhD.

  * •

Age was taken from the demographics form.

  * •

Platform, either LabintheWild, or Prolific. LabintheWild was used for
participants that organically found and took the study through LabintheWild
platform. While Prolific was used for participants that took the experiment
for paid compensation using the Prolific platform.

Additionally, metrics for gender, country of origin, reasons for data
exclusion (e.g., self-reported cheating), and indication of technical issues
were recorded from the demographics form and the self-reported questionnaire
at the end of the experiment. Devices and browsers used by the participants
were also logged after participants agreed to the consent form.

####  4.4.2. Article Properties

Article metrics included:

  * •

Article Id, a unique ID–one for each of the 94 articles

  * •

Article Time corresponds to the time taken by a participant to read or view an
article444before they clicked a blue arrow to see the next slide which
included the first question for that article.

  * •

Article Knowledge, the answer to a five-nominal scale familiarity question
regarding the topic of the article. The scale categories were: “I’ve never
heard of it”, ‘I’ve heard of it but I don’t know what it is”, ‘I have some
knowledge about this topic”, ‘I know a lot about this topic”, and ‘I’m an
expert on this topic”.

  * •

Article Word Count for each of the 94 articles in our dataset.

  * •

Image Presence a binary variable corresponding to the presence/absence of an
image in an article viewed by a participant.

  * •

Readability was determined using the Flesch-Kincaid grade-level metric.

####  4.4.3. Question Properties

We record the following data for questions in our experiment:

  * •

Question Type identifies the type of question the participant saw: general,
visual, or image.

  * •

Individual Question Score, a binary variable that identifies whether the
question was answered correctly or not.

  * •

Question Time, the difference between timestamps of when the participant first
saw the question and when they clicked a blue arrow to see the next slide in
the experiment.

  * •

Visual Knowledge Question Consistency, an indicator (only for visual knowledge
questions) of whether the question could be answered by looking at the image
alone.

###  4.5. Models

To understand how images impact participant question accuracy, we fitted a
series of linear mixed-effects models using the data collected from the main
experiment and the lme4 R package (Bates et al., 2015). To train these models,
we excluded data from the 101 participants who chose not to provide optional
demographic information such as age, gender, education level, and English
proficiency. This resulted in the models using data from 603 participants. We
separated our data into three sets (i.e., by “general”, “visual”, and “image”
recognition question responses) and trained three separate full and null
models using the separated question type data sets.

First, the null models (intercept/empty) were fitted with Article Id and
Participant Id as random factors and Individual Question Score as the
dependent variable. Next, we created full models with Individual Question
Score as the dependent variable, and we used the following as independent
variables: Article Time, Question Time, Article Knowledge, Age, Article Word
Count, English Proficiency, Education Level, Platform, and Image Presence.
Again, Article Id and Participant Id were used as random factors. The visual
question model includes an extra independent variable of Visual Knowledge
Question. We used the null and full model created for each data set category
(“general”, “visual”, and “image”) to perform comparison using the likelihood
ratio test.

Each full model produced unique results showing how distinct parts of the
experiment impacted participants’ success in answering the three different
question types. These models were built on limited data and our main focus was
not on achieving better model prediction. Instead, we wanted to identify
variables within our experiment that influenced participants’ knowledge
acquisition which we gauged through question accuracy. Section 5 further
explores our results, showing that each full model for the different question
types contains a marginal $R^{2}$ below $.05$. Clearly, there may be other
uncaptured variables that better predict question accuracy. Nonetheless, these
models give relevant insights into the variables that affected participants in
our experiment.

##  5\. Results

Our results show that images in Wikipedia articles are only helpful in some
cases of knowledge acquisition. The presence of an image did not significantly
affect the accuracy of the responses if participants were asked general and
visual knowledge questions. However, it significantly improved the accuracy of
the responses to the image recognition questions.

###  5.1. Baseline and (No) Image Performance

To establish whether learning could take place, and how much, we began by
analyzing the data from our baseline experiment (see 4.1). The average test
score for each type of question, without exposure to article text nor images,
indicates both how well an ‘average’ participant knows the material and how
guessable/inferable the correct answer is. Baseline accuracy was 44%, 47%, and
59% for the general knowledge, visual knowledge, and image recognition
questions respectively (see Figure 6). This is higher than a random guess
(25%) but does not create a ceiling effect.

As can be seen in Figure 6, any intervention (text or text with image),
results in significant improvements over baseline for knowledge acquisition
and retention ($F_{(1)}=60.91,p<.0001$). Including an image with the article
helps with the image recognition questions (73% versus 59%). Interestingly,
with text alone, participants did not perform much better than the baseline
for these questions (60% versus 59%). This supports the critical importance of
images for some learning tasks.

On initial analysis, images did not appear to help with general knowledge
questions and may hurt performance slightly (though the difference is not
significant). With general knowledge questions, there is no ‘knowledge’
benefit to the images, so this result is perhaps expected. However, there is
always a concern that images may be a distraction and lead to less engagement
with the text. More surprisingly, there were no differences in the visual
knowledge questions with and without the image (66%). We expand on these
results below.

Figure 6. All question types broken into the image + text condition (“Image”)
and the text-only conditions (”No Image”). These are compared to baseline
performance (blue line) when neither text nor image are provided.

###  5.2. Do images help learning? (RQ1)

To test the impact of images in question accuracy, we applied regression
models using various metrics collected for individuals, questions, and
articles. Below we describe each of the question type models in more detail.

####  5.2.1. General Type Question Model

A comparison of the null model with the full general question type model using
the likelihood ratio test showed that the full model fits the data
significantly better ($\chi^{2}_{(15)}=2593.7$, $p<.0001$). Our General
Knowledge Question Model only looked at the question data collected from
general knowledge questions and explains 19% of participants’ performance on
these specific questions (conditional $R^{2}=.19$, marginal $R^{2}=.034$). As
can be seen in Fig. 7(a), the presence of the image does not significantly
improve the question score for the general knowledge question type.
Specifically, the model indicates that the longer participants read or viewed
the article, the better they performed on the general knowledge question
($F_{(1)}=16.93,p<.001$). Participants also did better on general knowledge
questions when they indicated that they had a higher level of education
($F_{(1)}=9.24,p<.05$) and English proficiency ($F_{(1)}=9.17,p<.05$).
Readability level and image presence did not have a statistically significant
effect on knowledge comprehension for general knowledge questions.

(a) Mixed effect model predicting general knowledge question type accuracy.
Adjusted $R^{2}=.19$.

(b) Mixed effect model predicting visual knowledge question type accuracy.
Adjusted $R^{2}=.209$.

(c) Mixed effect model predicting image recognition question type accuracy.
Adjusted $R^{2}=.221$.

Figure 7. Results for the (a) General Knowledge, (b) Visual Knowledge, and (c)
Image Recognition question type models

####  5.2.2. Visual Knowledge Question Type Model

A comparison of the null model with the full visual knowledge question type
model using the likelihood ratio test showed that the full model fits the data
significantly better ($\chi^{2}_{(15)}=2678.0$, $p<.0001$). We created the
Visual Knowledge Question model using data collected for the visual knowledge
question type. The model explains 20.9% of participants’ performance
(conditional $R^{2}=.209$, marginal $R^{2}=.025$). The model shows that
question time, article time, English proficiency, and visual knowledge
question consistency affect how well people did on visual knowledge questions.
The longer the time that participants took to answer the visual knowledge
question, the lower their question score ($F_{(1)}=9.31,p<.05$) as seen in
Fig. 7(b). Similarly to the General Knowledge Question Type Model, the longer
participants took viewing the given article the better they did on the visual
knowledge question as well ($F_{(1)}=8.84,p<.05$). This model also indicated
that English proficiency had a significant main effect on participant accuracy
($F_{(1)}=4.3,p<.05$). Similarly to general knowledge questions, readability
level and image presence had no statistically significant effect on visual
knowledge questions. For visual knowledge questions, we included an additional
variable of visual consistency. In the resulting model, the variable indicates
that visual knowledge questions that could be answered with the help of the
randomly shown image (i.e., were consistent) yielded better performance
($F_{(1)}=8.14,p<.05$).

####  5.2.3. Image Recognition Question Type Model

A comparison of the null model with the full image recognition question type
model using the likelihood ratio test showed that the full model fits the data
significantly better ($\chi^{2}_{(15)}=2513.5$, $p<.0001$). Image Recognition
Question Type Model that only looked at data for the image recognition
question type resulted in explaining 22.1% of participant’s question score on
the image recognition question (conditional $R^{2}=.221$, marginal
$R^{2}=.048$). Both lower question time ($F_{(1)}=20.24,p<.001$) and longer
article time ($F_{(1)}=13.41,p<.001$) increased participant knowledge
comprehension as seen in Fig. 7(c). Out of all the models, this is the only
one where the presence of the image significantly impacted the accuracy of the
participants. With images present, participants performed significantly better
on the image recognition question ($F_{(1)}=24.06,p<.001$).

Recall that for all experiments we treated the first article as a practice
round and excluded it from analysis (Section 4.3). Notably, after removing the
first article data, our model remained largely unchanged for all question
types with one key exception: the platform became significant. Individuals on
the Prolific platform appeared to do worse overall. We hypothesize that this
may have to do with motivation. LabintheWild participants, who were more
intrinsically motivated, may be engaged from the first article and question.
Prolific participants may optimize largely for specific task success. Once
they have learned the task, they will focus on reading to achieve better
results. Future work may validate this hypothesis or provide alternative
explanations.

###  5.3. Which images help learning? (RQ2)

During which period was Newgrange built?

When was the Worldwide Plaza completed?

When was ”La Scapigliata” painted?

(a) General knowledge questions where images help with content retention.

What is the material of Donatello’s David?

How tall can totem poles be?

(b) Visual knowledge questions where images help content retention.

Figure 9. Qualitative analysis–examples of images that are helpful to answer
generic questions. (a) Visual arts characteristics for general knowledge
questions. (b) Visually consistent images for visual knowledge questions.

The literature studying the role of images in instructional settings suggests
that the impact of images on knowledge acquisition depends, among other
things, on the visual characteristics of the image (extensively surveyed by
Clark et al. (Clark and Lyons, 2010)).

To understand whether the trends outlined above were consistent across
different image types, we performed a manual inspection of the images whose
presence generated greater positive/negative differences in the question
scores.

####  5.3.1. General knowledge questions

Our analysis shows that images largely had no significant impact on general
knowledge question accuracy. To expose interesting patterns of exception to
this trend, we inspected our results more in-depth. For each question, we have
a question score that is either 0 if the question was answered correctly (1,
otherwise). This makes scores quite comparable across questions of different
types. For each question, an image effect value was calculated, by averaging
the question score for all responses where image presence was true, and
subtracting the average question score for all responses where image presence
was false. The larger the image effect value, the larger the impact of the
image presence on the participant’s accuracy. Image effect scores ranged from
$-1$ for questions where image presence had negative impact, all the way to
$0.83$ for questions where images have a positive impact on accuracy. We found
that, as expected, the image effect score is 0 for the vast majority of
questions (65%), and that only a small number of questions have a non-zero
image effect, equally distributed across the positive and negative spectrum.
We then ranked general knowledge questions by image effect, and examined the
top 15 questions, articles, and images, for which the image had a positive
effect. We found that, in some cases, images were useful to answer general
knowledge questions for articles about visual arts and architecture. Images
were helpful to estimate epochs (i.e., time periods) and styles of paintings,
or the century when monuments were built (see Fig. 9(a)).

Figure 10. Visual Knowledge Question Compared to Baseline Performance with
Consistency Metric

####  5.3.2. Visual knowledge questions

As indicated by our high-level analysis, we saw no effect of images on visual
knowledge questions. Although we used only text to create visual knowledge
questions, when present, images should have been helpful in answering these
questions, especially those images that are consistent with the visual
descriptions in the text. To better understand the impact of images on visual
knowledge question accuracy we labeled each image in our dataset with a
“visual knowledge question consistency”, that indicated whether a visual
knowledge question could be answered using the article image (see Figure
9(b)). In Figure 10 we can see how visual knowledge question consistency
impacted accuracy in comparison to the baseline. The baseline accuracy between
consistent and not consistent visual knowledge questions is not significant
($F_{(1)}=2.8,p=.096$) based on existing limited data. However, consistency is
a significant main effect ($F_{(1)}=6.32,p<.05$) for the visual knowledge
question type data collected from our main experiment. Participants tended to
get correct answers to visual knowledge questions more often when consistent
images are displayed along with the article.

Low quality (Spectral Bat)

High quality (Leafy Seadragon)

(a) Image quality

Part (900 North Michigan)

Whole (900 North Michigan)

(b) Part VS Whole

Illustrated
version (Nepeta)

Photographed version (Nepeta)

(c) Photo VS Illustration

Figure 12. In-depth analysis - examples of images that are harmful to answer
the image question. (a) Low Quality images. (b) Images that describe only a
part of the object. (c) Photographed objects in the article vs illustrations
in the question.

####  5.3.3. Image recognition questions

In our quantitative results, we observed that participants who saw text+image
articles generally performed significantly better on image recognition
questions. To reveal possible exceptions to this trend, we performed the same
ranking procedure of general knowledge questions and inspected the 15
questions ranked low by image effect. The image effect scores ranged from
$-0.675$ for questions where image presence had negative impact, all the way
to $0.75$ for those questions where images have a positive impact on accuracy.
Notably, the negative image effect was for a relatively small proportion of
image questions ($25\%$). We noticed that when the original article image is
of poor quality 555To evaluate image quality, we checked whether images fit
the image quality criteria from the manual of style on Wikipedia(Wikipedia,
2022a). These include basic photographic quality rules, such as exposure and
clutter, as well as more generic rules to optimally represent the article’s
subject. (see Figure 12(a)), participants struggled with image recognition
questions. Also, we noticed lower accuracy when article images depicted the
subject as a whole, while image recognition questions depicted only a part or
detail of the subject, or vice versa (see Figure 12(b)), or when image
recognition questions depicted the subject in form of illustration, while the
article image was a photographic rendering of the subject (see Figure 12(c)).
Image presence had no significant impact on image recognition question
accuracy when the subject of the article was inherently difficult to
distinguish visually, for example, for the article about Victoria Falls.
Finally, participants who saw illustrated articles failed when image
recognition questions were trickier than average. Harder questions included:
“Which of these is a male upland goose?”, or “Which of these is not the Taj
Mahal?”

##  6\. Discussion and Conclusions

Wikipedia is enormously important for helping people learn about a variety of
topics. Images from Wikipedia are an integral part of this experience, both on
Wikipedia and where the content is repurposed (e.g., search engine result
pages). Despite the many “views” Wikipedia images get every day, we know
little of the role that images play in people’s information uptake. In this
work, we contribute an open-source curated question & answer dataset to test
knowledge comprehension and recall in Wikipedia. Additionally, our study makes
several empirical contributions:

##### Learning about general knowledge is rarely supported by images.

We found that images do not always help learning about general knowledge facts
of a topic—in fact, participants were as likely to correctly answer general
knowledge questions when the Wikipedia article did not include an image as
they were when there was an image. However, our experiment revealed specific
cases where images did support answering general knowledge questions (see Fig.
9(a)). For example, remembering when a building was built or a painting was
created may be easier with an image if the viewer can make inferences based on
how that building or painting looked. If a viewer generally recalls the
properties of a Romantic painting, they may be able to accurately estimate
when a painting was made (even without seeing the date in text). For some
topics (and with some background knowledge), images on Wikipedia may help
readers create a mental model of an object that they can then use for
inferences (Gyselinck and Tardieu, 1999). Future work may more generally
identify when such images help and why.

##### Learning about visual knowledge is only supported by images if they are
consistent with the text.

Interestingly, the presence of images did not significantly improve answer
accuracy for visual knowledge questions, such as questions about materials,
color, or size. In fact, having an image or not resulted in the same answer
accuracy in our study, suggesting that images, in general, do not support
visual knowledge learning. This is perhaps the most surprising result of our
study, as images are commonly assumed to aid engagement and learning (Clark
and Lyons, 2010). However, this result is not entirely inconsistent with past
work. Related prior studies on image use in more general online contexts has
also found that images may not support content retention (Beymer et al.,
2007).

In our follow-up analysis, we found that images that are consistent with their
visual knowledge questions are answered accurately significantly more often
than images that are inconsistent with the question. Images seem to boost
answer accuracy when they are closely aligned with the text. This is in line
with prior work on misaligned or ‘seductive’ images (e.g., (Harp and Mayer,
1997; Peeck, 1993)). Guides for teachers recognize this as the first of the 10
recommendations for pictorial facilitation of text information (the “10
tenents for teachers”) from Carney and Levin (Carney and Levin, 2002): “Select
pictures that overlap with text content”. However, in our visual consistency
annotations, we found that less than 40% of the image-visual knowledge
question pairs in our dataset are aligned. Since visual knowledge questions
are generated to reflect the main (and potentially more interesting) features
described in the article text, this suggests that many images in Wikipedia are
not optimally chosen to reflect the textual content in the article. While not
all visual characteristics are equally important, depending on the learning
context, our initial experiment suggests a number of hypotheses for future
study. For example: that Wikipedia editors may find it difficult to anticipate
what makes a good image; that no single image can illustrate all important
characteristics; or that the Wikimedia Commons does have great images choices
to pick from.

##### Images help visual recognition.

We found that images generally enhance the ability of readers to visually
recognize other instances of the object described in an article, compared to
relying only on textual information. These findings suggest that images can be
quite useful in encyclopedic settings, since the ability to visually recognize
and classify objects is a fundamental aspect of everyday life (Bruce and
Humphreys, 1994). Unillustrated articles, while providing enough useful
information to learn about general aspects of a subject, do not offer a full
visual description. This may naturally affect the reader’s ability to
recognize objects of that class or instance. In fact, what determines the
accuracy of our image recognition questions is not whether the participant
read the article (image recognition question scores are the same for the
baseline and experiment conditions), but whether an image was present when the
articles were read.

These results are consistent with theories of images helping with retention.
For example, dual coding (Paivio, 1990) has been applied in educational
contexts to justify the use of images (Clark and Lyons, 2010; Vekiri, 2002)
and identify their potential limitations (Harp and Mayer, 1997). While there
is evidence in our results that Wikipedia images may not be optimally chosen,
it is important for future work to reflect on what we are optimizing for. Our
choice of visual knowledge questions was driven by the specific article
text—that is, aspects of category or instance that were judged important
enough to be included in the article. While these were not ‘outlier’ questions
(e.g., one in one million birds is red instead of blue), we did not rank the
facts/questions based on importance.

The idea of ‘importance’ connects to various theories of categories. For
example, we could consider the classic linguistic notion of necessary and
sufficient properties by which we would classify an object into a category.
These theories posit that it is possible to create categorical boundaries and
hierarchies through this classification. Those properties (e.g., the unique
shape of a bird’s wing and coloring) may be the most crucial to emphasize both
in text and in an image. However, it is not clear that all items in Wikipedia
could be reduced to this model. Wikipedia, and our dataset, reflect pages for
both classes (e.g., Totem Poles) and instances (the painting ‘La
Scapigliata’). For the latter, it is not obvious that classifying an item,
such as a painting, by either things that make it unique or things that put it
in a category, is appropriate. We may simultaneously want to understand what
makes an item look unique and look similar or different to other related items
(e.g., other paintings from the time period). This makes the selection of both
‘key facts’ and good image challenging. Prototype theory (Rosch, 1973)
reflects an alternative framework, though largely a linguistic one, in that
one can judge how close categories are to each other. For example, one could
argue that a robin is a better prototype of a bird than is a penguin (based on
general understanding). By extension, the characteristics of a robin are a
better representation of the bird category. Insofar as we can extend this to
visual properties, one could ask ‘how much would you associate a visual fact
(e.g., the blue coloring) with the category or instance?’ We note that while
we did not explicitly seek prototypical descriptions when crafting questions,
we did avoid those that might only apply to outliers. However, it should be
emphasized that prototypical visual characteristics may not be the most
important ones for a learning task. For example, we might care about non-
prototypical characteristics when describing the similarity of two things. As
an area of future work, we might use domain experts to re-rank facts, and
therefore questions, by how ‘important’ they are to the teaching of a concept.
This would allow for a more refined analysis of the (in)consistent image
question rather than a simple binary one. Thus, we could answer not just the
question, ‘does this image express a particular characteristic or set of
characteristics from the text?’ but also ‘how well does this image meet our
specific learning objectives?’

##### Photographic quality matters.

The cognitive theory of multimedia learning suggests that higher quality
visuals reinforce and extend knowledge (Mayer, 2002). Upon manual inspection,
we found that there might be some relation between photographic quality and
knowledge acquisition from Wikipedia articles. We found that images of low
photographic quality (see, for example, Figure 12(a)) can hinder participants’
ability to visually recognize the subject of the article. Guidelines for image
use in field guides–books used to recognize and research wildlife–suggest that
high-quality images are essential to make good educational material,
especially when it comes to introductory sections and glossaries (Leggett and
Kirchoff, 2011). We also found that people may find it difficult to generalize
the insight from an image to other instances. For example, when shown an
illustrated version of a plant, our participants were unable to accurately
recognize the same plant in a photographed version. This could indicate that a
dual representation is helpful, especially for article topics that are sought
for learning what something looks like. It also indicates that Wikipedia
images are not always chosen to include representative features to be
perceived as being of the same category.

###  6.1. Design Implications

Collectively, our findings have several theoretical and practical
implications.

##### Promote the addition of missing images on Wikipedia

Wikiwork—human contributions that improve Wikipedia in various ways—is
extremely varied. Some editors add new articles, while others focus on
improving citations or correcting grammar mistakes. Unfortunately, it is not
obvious that image-focused wikiwork is as widely understood or appreciated. We
see a reflection of this in the Barnstars awarded by the Wikipedia community
to editors (Kriplean et al., 2008; Wikimedia Foundation, Inc., 2022f).
Barnstars can be awarded for everything from diligence to being a ‘surreal
contributor’ (“any Wikipedian who adds ‘special flavor’ to the community by
acting as a sort of wildcard”). The range of these awards provide a partial
signal of what kind of work is valued. While there are hundreds of types of
Barnstars only a handful are dedicated to any kind of image work. For example,
an SVG Barnstar is awarded to those converting raster to vector images. A
Valued Picture Barnstar is awarded to, “users that uploaded a free image to
Wikipedia that is considered wonderful, valuable, and/or crucial” and the
Photographer Barnstar goes to users “who tirelessly improve Wikipedia with
their photographic skills and contributions.” (Wikimedia Foundation, Inc.,
2022f).

Our work demonstrates the importance of good images for a variety of learning
objectives. As learning represents a key use case for Wikipedia (Singer et
al., 2017), wikiwork focused on curating, finding, or creating good images
should be more valued. Although Barnstars alone are unlikely to provide
motivation on their own, acknowledging work is a key component in increasing
editor retention and norm-setting (Matias et al., 2020). Exactly what this
kind of image-focused contribution looks like, and what types should be
prioritized, is an extensive topic for research. However, these measures might
be included within community-led efforts to add images to unillustrated
Wikipedia articles, such as the already existing Wikipedia Pages Wanting
Photos #WPWP campaign (Campaigns, 2022), or the Visible Wiki Women
(Knowledge?, 2022) initiative, as well as those “Wiki Loves” campaigns
designed to improve the visual coverage of encyclopedic topics, such as
monuments (Wiki Loves Monuments (Monuments, 2022)), or natural heritage (Wiki
Loves Earth (Earth, 2022)). In addition to ‘novel’ types of wikiwork, image-
centered contributions can also become integrated into existing structures.
For example, editors within a subtopic (e.g., birds, sociology, food, etc.)
will often agree on templates and structures for those pages around text or
infoboxes. Such constraints are difficult for images. For example, we may want
all bird pages to have a consistent look and feel for their images, but the
perfect image may not exist (or may not be in fair-use). While subject-matter
experts disagree over whether photographs or paintings/illustrations are best
for characterizing the world (Sutton, 2021; Leggett and Kirchoff, 2011), there
is general agreement that a consistent look is important. This presents a
unique and interesting sociotechnical challenge. The best interventions to
normalize and prioritize image-centered wikiwork, all within the unique
‘bureaucracy’ of Wikipedia (Butler et al., 2008), present important future
topics for research.

##### Promote the addition of missing images to Wikidata

The findings in our paper also highlight the importance of adding high-quality
images to Wikidata. Wikidata is a Wikimedia project and the open, manually
curated knowledge graph that allows one to connect all Wikipedia articles
across different languages. For example, the Train article exists in 162
languages, that often inherit structured information and the infobox image
from the corresponding Wikidata item (Wikidata, 2022), either manually or
automatically. Such an image may not be ‘best’ or even ‘good’ and may not be
culturally appropriate. While highly edited languages may counteract this kind
of behavior and replace the image, languages with lower editorial resources
may not.

##### Encourage the addition of images about historic eras

Our findings suggest that articles that describe aspects of historical time
periods, such as specific paintings or architectural artifacts of a certain
time, may benefit from accompanying images. While architecture and visual arts
figure among the top topics by image presence on Wikipedia (Wikimedia
Foundation, Inc., 2022c), a large percentage of these articles still remain
unillustrated. A concrete step building on our results could be to promote
initiatives that encourage articles about such topics include high-quality
images. Moreover, existing editor groups who work on improving Wikimedia
content on these topics, such as WikiProject Cultural Heritage on Wikidata
(Wikidata, 2022), or Wikiproject Historic Sites on Wikipedia (Wikipedia,
2022b), could explicitly encourage contributions to the visual content of the
articles and items in these domains.

##### Encourage or suggest images that are consistent with the text.

While Wikipedia editors can freely choose what images to add, our findings
show that the chosen images are not always optimal. In fact, our analysis
showed that images that are relevant to the text lead to higher accuracy in
answering visual knowledge questions. Wikipedia’s manual of style (Wikipedia,
2022a) already suggests to choose consistent images, but our dataset shows
that this is not always followed. While the manual of style speaks to the
importance of image relevance to the generic “topic’s context”, we find here
that images that are not aligned with the visual description of the subject
can hinder the ability to learn. Additional guidelines and examples may be
needed for ensuring that images are consistent with descriptive text and to
alert editors of the negative impact of inconsistent images. Additionally,
existing multimodal machine learning frameworks that measure the alignment
between images and text (Radford et al., 2021; Srinivasan et al., 2021), could
be incorporated into image suggestion frameworks to select the best images for
unillustrated articles, such as the add-an-image task for newcomer editors
(Wikimedia Foundation, Inc., 2022a).

##### Discourage low-quality images.

Our study also reveals that images of low quality can hinder participants’
ability to visually recognize the subject of the article. The English
Wikipedia manual of style (Wikipedia, 2022a) indeed suggests that images of
“Poor-quality images—dark or blurry; showing the subject too small, hidden in
clutter, or ambiguous; and so on—should not be used unless absolutely
necessary”. Automated image quality classifiers (Beytía et al., 2022) have
been trained based on images labeled as “Quality Images” (Wikimedia Commons,
2022) by the Wikimedia Commons community. Given resource limitations on good
fair-use images, creating images may be crucial. Selecting culturally
appropriate images for different Wiki language domains may also be an
important form of labor. To promote the usage of high quality images, similar
computer vision-based classifiers could be incorporated into image suggestion
algorithms, or search tools, that help editors find the right images for
Wikipedia articles.

###  6.2. Limitations and Future Work

We hope that our manually curated question & answer dataset to test knowledge
comprehension in English Wikipedia can be used and extended for other studies
of images on Wikipedia. For example, our current dataset is focused on showing
one specific image alongside the text (the so-called ‘lead’ image). However,
there is no evidence that this is the best image of a concept that Wikipedia
or the Commons has to offer. A natural extension would be to experimentally
vary the image displayed alongside the text with others that are available in
the Commons. Such a use would provide guidance for editors, and potentially
tool builders, in identifying not just a reasonable image, but the best one
among a set.

A limitation of our dataset is that our questions were manually generated by a
small group of researchers. Although we asked the annotators to follow a
thorough process and take into account the interestingness of the questions
for the general public, the pool of questions in our data might not reflect
the actual information needs and interests of Wikipedia readers. Future
research is needed to align our reading comprehension questions with readers’
motivations, and to analyze how images are useful to satisfy the different
information needs.

Moreover, while the dataset is the first of its kind, its scale can be further
expanded by automating article, section, image, and question retrieval using
Natural Language Processing to detect Wikipedia articles which include a
physical description and automatically generate questions that can be verified
in crowdsourcing settings. Such larger datasets would enable larger-scale
experiments, which could make use of automated analysis of visual content to
determine which subjects, spatial arrangements, and image types are more
useful to improve reading comprehension.

Alignment between images and their corresponding captions also plays an
important role in text comprehension (Bernard, 1990). This topic even has a
dedicated page in Wikipedia’s manual of style (Wikipedia, 2022a). While
investigating the importance of captions in Wikipedia knowledge acquisition
was outside the scope of this work, this is an area for future research which
could be implemented with the data and framework provided with this study.

Our study is also limited by being offered only in English. Our findings may
not generalize to other languages and people from other countries and
cultures. Future work could extend our study to a more diverse population to
study the helpfulness of images with a sample more representative of Wikipedia
users.

In this initial work, we have focused on both a set of assumed motivations and
learning objectives for reading Wikipedia. Although we believe that our
choices are a good proxy for common tasks on Wikipedia, they do not represent
the full breadth and complexity of the space. Additional research, with more
types of images, articles, levels of expertise, learning objectives,
assessments, and so on, will further clarify which images are contextually
useful.

##  7\. Conclusion

In this work, we set out to evaluate whether images on Wikipedia support a set
of common learning tasks. Our results build on research to understand the
impact of text and media on learning. Wikipedia is a particularly important
platform to analyze due to its broad impact and unique crowdsourced
organization.

Using a novel dataset of article excerpts, images, and questions we conducted
a large-scale online experiment (n=704) with participants on LabintheWild and
Prolific. Our results reveal the specific utility of images on different types
of questions. Encouragingly, we did not find situations where images markedly
harm learning tasks. For image recognition tasks, article images significantly
improved reader accuracy in selecting target concepts (e.g., _which of these
pictures is a leafy seadragon?_). We also found that for visual knowledge
questions, only a subset of images help. Given the immense reach of Wikipedia
images, our results have broad implications for policy, design guidelines, and
novel tools.

## Acknowledgements

We are grateful to the NSF for their support of this work through NSF
IIS-1815760 and NSF IIS-2006104.

## Image Attributions

Figure 1:

  * •

Leafy Seadragon on Kangaroo Island.jpg; James Rosindell, CC BY-SA 4.0

Figure 2:

  * •

2005-09-20 1080x1920 chicago 900 north michigan.jpg; J. Crocker, Attribution

Figure 5 (c):

  * •

Gujia dry.JPG; Prasant iitd, CC BY-SA 4.0

  * •

Fried dumpling (3337901416).jpg; pelican from Tokyo, Japan, CC BY-SA 2.0

  * •

Xia jian jiao.jpg; lazy fri13th, CC BY-SA 2.0

  * •

Deep Fried Cholent Dumpling (Fülemüle Restaurant)
(41496749174).jpg; Asok5 from Budapest, Hungary, CC BY-SA 2.0

Figure 8 (a):

  * •

Irelands history.jpg; Tjp finn, CC BY-SA 4.0

  * •

Wwp50roof.JPG; Jim.henderson at the English Wikipedia, Public domain

  * •

Leonardo da vinci - La scapigliata.jpg; Leonardo da Vinci, Public domain

Figure 8 (b):

  * •

Florenz - Bargello 2014-08-09r.jpg; Rabe!, CC BY-SA 4.0

  * •

Two Brothers totem pole Jasper.jpg; Ymblanter, CC BY-SA 4.0

Figure 10 (a):

  * •

Spectral bat photo.jpg; Marco Tschapka, CC BY-SA 3.0

  * •

Leafy Seadragon on Kangaroo Island.jpg; James Rosindell, CC BY-SA 4.0

Figure 10 (b):

  * •

2005-09-20 1080x1920 chicago 900 north michigan.jpg; J. Crocker, Attribution C

  * •

hicago 900 North Michigan Avenue (2827940356553).jpg; Roman Boed from The
Netherlands, CC BY 2.0

Figure 10 (c):

  * •

Calamintha nepeta Sturm54.jpg; Johann Georg Sturm (Painter: Jacob Sturm),
Public domain

  * •

Catnip-blossom.jpg; Jon Sullivan, Public domain

## References

  * (1)
  * Adler et al. (2008) B. Thomas Adler, Luca de Alfaro, Ian Pye, and Vishwanath Raman. 2008.  Measuring Author Contributions to the Wikipedia. In _Proceedings of the 4th International Symposium on Wikis_ (Porto, Portugal) _(WikiSym ’08)_. Association for Computing Machinery, New York, NY, USA, Article 15, 10 pages.  https://doi.org/10.1145/1822258.1822279
  * Anderson et al. (2001) L.W. Anderson, D.R. Krathwohl, and B.S. Bloom. 2001\.  _A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives (Complete Edition)_.  Longman.
  * August and Reinecke (2019) Tal August and Katharina Reinecke. 2019.  Pay attention, please: Formal language improves attention in volunteer and paid online experiments. In _Proceedings of the 2019 chi conference on human factors in computing systems_. 1–11.
  * Bates et al. (2015) Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015.  Fitting Linear Mixed-Effects Models Using lme4.  _Journal of Statistical Software_ 67, 1 (2015), 1–48.  https://doi.org/10.18637/jss.v067.i01
  * Bernard (1990) Robert M Bernard. 1990\.  Using extended captions to improve learning from instructional illustrations.  _British Journal of Educational Technology_ 21, 3 (1990), 215–225.
  * Beymer et al. (2007) David Beymer, Peter Z Orton, and Daniel M Russell. 2007\.  An eye tracking study of how pictures influence online reading. In _IFIP Conference on Human-Computer Interaction_. Springer, 456–460.
  * Beytía et al. (2022) Pablo Beytía, Pushkal Agarwal, Miriam Redi, and Vivek K Singh. 2022.  Visual gender biases in wikipedia: A systematic evaluation across the ten most spoken languages. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 16. 43–54.
  * Bloom et al. (1956) Benjamin S Bloom, Max D Engelhart, Edward J Furst, Walker H Hill, and David R Krathwohl. 1956\.  _Taxonomy of educational objectives, handbook I: The cognitive domain_. Vol. 19.  New York: David McKay Co Inc.
  * Bruce and Humphreys (1994) Vicki Bruce and Glyn W Humphreys. 1994.  Recognizing objects and faces.  _Visual cognition_ 1, 2-3 (1994), 141–180.
  * Butler et al. (2008) Brian Butler, Elisabeth Joyce, and Jacqueline Pike. 2008\.  Don’t look now, but we’ve created a bureaucracy: the nature and roles of policies and rules in wikipedia. In _Proceedings of the SIGCHI conference on human factors in computing systems_. 1101–1110.
  * Campaigns (2022) Wikimedia Campaigns. 2022\.  Wikipedia Pages Wanting Photo Campaign.  https://meta.wikimedia.org/wiki/Wikipedia_Pages_Wanting_Photos_2022
  * Carney and Levin (2002) Russell N Carney and Joel R Levin. 2002.  Pictorial illustrations still improve students’ learning from text.  _Educational psychology review_ 14, 1 (2002), 5–26.
  * Clark and Lyons (2010) Ruth C Clark and Chopeta Lyons. 2010.  _Graphics for learning: Proven guidelines for planning, designing, and evaluating visuals in training materials_.  John Wiley & Sons.
  * Diego Saez-Trumper (2022) Diego Saez-Trumper. 2022\.  Why We Read Wikipedia Data.  https://medium.com/diegosaeztrumper/open-data-and-covid-19-wikipedia-as-an-informational-resource-during-the-pandemic-dcca6a23e826
  * Dimitrov et al. (2016) Dimitar Dimitrov, Philipp Singer, Florian Lemmerich, and Markus Strohmaier. 2016.  Visual positions of links and clicks on wikipedia. In _Proceedings of the 25th International Conference Companion on World Wide Web_. 27–28.
  * Earth (2022) Wiki Loves Earth. 2022\.  Wiki Loves Earth Campaign.  https://wikilovesearth.org/
  * English Wikipedia (2021) English Wikipedia. 2021\.  Wikipedia:Unusual articles.  https://en.wikipedia.org/wiki/Wikipedia:Unusual_articles
  * English Wikipedia (2022) English Wikipedia. 2022\.  Leafy Seadragon.  https://en.wikipedia.org/wiki/Leafy_seadragon
  * Erickson et al. (2018) Kristofer Erickson, Felix Rodriguez Perez, and Jesus Rodriguez Perez. 2018.  What is the Commons Worth? Estimating the Value of Wikimedia Imagery by Observing Downstream Use. In _Proc. International Symposium on Open Collaboration (OpenSym)_.
  * Evenstein Sigalov and Nachmias (2017) Shani Evenstein Sigalov and Rafi Nachmias. 2017.  Wikipedia as a platform for impactful learning: A new course model in higher education.  _Education and information technologies_ 22, 6 (2017), 2959–2979.
  * Good et al. (2010) Jessica J Good, Julie A Woodzicka, and Lylan C Wingfield. 2010\.  The effects of gender stereotypic and counter-stereotypic textbook images on science performance.  _The Journal of social psychology_ 150, 2 (2010), 132–147.
  * Guo et al. (2020) Daibao Guo, Shuai Zhang, Katherine Landau Wright, and Erin M McTigue. 2020.  Do You Get the Picture? A Meta-Analysis of the Effect of Graphics on Reading Comprehension.  _AERA Open_ 6, 1 (2020), 2332858420901696\.
  * Gyselinck and Tardieu (1999) Valérie Gyselinck and Hubert Tardieu. 1999.  _The role of illustrations in text comprehension: What, when, for whom, and why?_ Lawrence Erlbaum Associates Publishers, Mahwah, NJ, US, 195–218.
  * Haladyna (2004) Thomas M Haladyna. 2004\.  _Developing and validating multiple-choice test items_.  Routledge.
  * Harp and Mayer (1997) Shannon F Harp and Richard E Mayer. 1997.  The role of interest in learning from scientific text and illustrations: On the distinction between emotional interest and cognitive interest.  _Journal of educational psychology_ 89, 1 (1997), 92\.
  * He et al. (2018) Shiqing He, Allen Yilun Lin, Eytan Adar, and Brent J Hecht. 2018\.  The_Tower_of_Babel. jpg: Diversity of Visual Encyclopedic Knowledge Across Wikipedia Language Editions.. In _Proc. International Conference on Web and Social Media (ICWSM)_.
  * Head and Eisenberg (2010) Alison Head and Michael Eisenberg. 2010.  How today’s college students use Wikipedia for course-related research.  _First Monday_ 15, 3 (2010).
  * Heald et al. (2015) Paul Heald, Kristofer Erickson, and Martin Kretschmer. 2015\.  The Valuation of Unprotected Works: A Case Study of Public Domain Images on Wikipedia.  _Harv. JL & Tech._ 29 (2015), 1.
  * Johnson et al. (2021) Isaac Johnson, Martin Gerlach, and Diego Sáez-Trumper. 2021\.  Language-agnostic topic classification for wikipedia. In _Companion Proceedings of the Web Conference 2021_. 594–601.
  * Kittur and Kraut (2008) Aniket Kittur and Robert E. Kraut. 2008.  Harnessing the Wisdom of Crowds in Wikipedia: Quality through Coordination. In _Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work_ (San Diego, CA, USA) _(CSCW ’08)_. Association for Computing Machinery, New York, NY, USA, 37–46.  https://doi.org/10.1145/1460563.1460572
  * Knight and Pryke (2012) Charles Knight and Sam Pryke. 2012.  Wikipedia and the University, a case study.  _Teaching in higher education_ 17, 6 (2012), 649–659.
  * Knowledge? (2022) Whose Knowledge? 2022\.  Visible Wiki Women Campaign.  https://whoseknowledge.org/initiatives/visiblewikiwomen/
  * Kriplean et al. (2008) Travis Kriplean, Ivan Beschastnikh, and David W. McDonald. 2008\.  Articulations of Wikiwork: Uncovering Valued Work in Wikipedia through Barnstars. In _Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work_ (San Diego, CA, USA) _(CSCW ’08)_. Association for Computing Machinery, New York, NY, USA, 47–56.  https://doi.org/10.1145/1460563.1460573
  * LabintheWild (2021) LabintheWild. 2021\.  Lab in the Wild.  https://labinthewild.org/
  * Laurent and Vickers (2009) Michaël R Laurent and Tim J Vickers. 2009.  Seeking health information online: does Wikipedia matter?  _Journal of the American Medical Informatics Association_ 16, 4 (2009), 471–479.
  * Lee-Robbins and Adar (2023) E. Lee-Robbins and E. Adar. 2023.  Affective Learning Objectives for Communicative Visualizations.  _IEEE Transactions on Visualization & Computer Graphics_ 29, 01 (jan 2023), 1–11.  https://doi.org/10.1109/TVCG.2022.3209500
  * Leggett and Kirchoff (2011) Roxanne Leggett and Bruce K. Kirchoff. 2011.  Image use in field guides and identification keys: review and recommendations.  _AoB PLANTS_ 2011 (01 2011).  https://doi.org/10.1093/aobpla/plr004 arXiv:https://academic.oup.com/aobpla/article-pdf/doi/10.1093/aobpla/plr004/17460593/plr004.pdf  plr004.
  * Lehmann et al. (2014) Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. 2014\.  Reader Preferences and Behavior on Wikipedia. In _Proc. Conference on Hypertext and Social Media (HT)_.
  * Lemmerich et al. (2019) Florian Lemmerich, Diego Sáez-Trumper, Robert West, and Leila Zia. 2019.  Why the world reads Wikipedia: Beyond English speakers. In _Proceedings of the twelfth ACM international conference on web search and data mining_. 618–626.
  * Levie and Lentz (1982) W Howard Levie and Richard Lentz. 1982.  Effects of text illustrations: A review of research.  _Ectj_ 30, 4 (1982), 195–232.
  * Li et al. (2019) Qisheng Li, Meredith Ringel Morris, Adam Fourney, Kevin Larson, and Katharina Reinecke. 2019\.  The impact of web browser reader views on reading speed and user experience. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Lucassen et al. (2012) Teun Lucassen, Roald Dijkstra, and Jan Maarten Schraagen. 2012\.  Readability of Wikipedia.  _First Monday_ (2012).
  * Marsh and White (2003) Emily E Marsh and Marilyn Domas White. 2003.  A taxonomy of relationships between images and text.  _Journal of Documentation_ 59, 6 (2003), 647–672.  https://doi.org/doi:10.1108/00220410310506303
  * Matias et al. (2020) J Nathan Matias, Reem Al-Kashif, Julia Kamin, Max Klein, and Eric Pennington. 2020.  Volunteers Thanked Thousands of Wikipedia Editors to Learn the Effects of Receiving Thanks.  https://citizensandtech.org/2020/06/effects-of-saying-thanks-on-wikipedia/
  * Mayer (2002) Richard E Mayer. 2002\.  Multimedia learning.  In _Psychology of learning and motivation_. Vol. 41. Elsevier, 85–139.
  * Monuments (2022) Wiki Loves Monuments. 2022\.  Wiki Loves Monuments Campaign.  https://www.wikilovesmonuments.org
  * Navarrete and Villaespesa (2020) Trilce Navarrete and Elena Villaespesa. 2020.  Image-based information: paintings in Wikipedia.  _Journal of Documentation_ (2020).
  * Paivio (1990) Allan Paivio. 1990\.  _Mental representations: A dual coding approach_.  Oxford University Press.
  * Peeck (1993) Joan Peeck. 1993\.  Increasing picture effects in learning from illustrated text.  _Learning and instruction_ 3, 3 (1993), 227–238.
  * Piccardi et al. (2020) Tiziano Piccardi, Miriam Redi, Giovanni Colavizza, and Robert West. 2020.  Quantifying Engagement with Citations on Wikipedia. In _Proc. The Web Conference (WWW)_.
  * Piccardi et al. (2021) Tiziano Piccardi, Miriam Redi, Giovanni Colavizza, and Robert West. 2021.  On the Value of Wikipedia as a Gateway to the Web. In _Proc. The Web Conference (WWW)_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International Conference on Machine Learning_. PMLR, 8748–8763.
  * Rama et al. (2022) Daniele Rama, Tiziano Piccardi, Miriam Redi, and Rossano Schifanella. 2022.  A large scale study of reader interactions with images on Wikipedia.  _EPJ Data Science_ 11, 1 (2022), 1.
  * Rello et al. (2016) Luz Rello, Martin Pielot, and Mari-Carmen Marcos. 2016.  Make it big! The effect of font size and line spacing on online readability. In _Proceedings of the 2016 CHI conference on Human Factors in Computing Systems_. 3637–3648.
  * Renkl and Scheiter (2017) Alexander Renkl and Katharina Scheiter. 2017.  Studying visual displays: How to instructionally support learning.  _Educational Psychology Review_ 29, 3 (2017), 599–621.
  * Rosch (1973) Eleanor H Rosch. 1973\.  Natural categories.  _Cognitive psychology_ 4, 3 (1973), 328–350.
  * Salutari et al. (2019) Flavia Salutari, Diego Da Hora, Gilles Dubuc, and Dario Rossi. 2019.  A Large-scale Study of Wikipedia Users’ Quality of Experience. In _Proc. International World Wide Web Conference (WWW)_.
  * Sayyadiharikandeh et al. (2019) Mohsen Sayyadiharikandeh, Jonathan Gordon, Jose-Luis Ambite, and Kristina Lerman. 2019.  Finding prerequisite relations using the wikipedia clickstream. In _Companion Proceedings of The 2019 World Wide Web Conference_. 1240–1247.
  * Schnotz and Bannert (2003) Wolfgang Schnotz and Maria Bannert. 2003.  Construction and interference in learning from multiple representation.  _Learning and Instruction_ 13, 2 (2003), 141–156.  https://doi.org/10.1016/S0959-4752(02)00017-8 External and Internal Representations in Multimedia Learning.
  * Singer et al. (2017) Philipp Singer, Florian Lemmerich, Robert West, Leila Zia, Ellery Wulczyn, Markus Strohmaier, and Jure Leskovec. 2017.  Why we read Wikipedia. In _Proceedings of the 26th international conference on world wide web_. 1591–1600.
  * Srinivasan et al. (2021) Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021\.  WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning.  _arXiv preprint arXiv:2103.01913_ (2021).
  * Sundararajan and Adesope (2020) NarayanKripa Sundararajan and Olusola Adesope. 2020.  Keep it coherent: A meta-analysis of the seductive details effect.  _Educational Psychology Review_ 32, 3 (2020), 707–734.
  * Sutton (2021) Clay Sutton. 2021\.  Can Photo Guides Do It All?  (2021).  https://www.aba.org/can-photo-guides-do-it-all/
  * Vekiri (2002) Ioanna Vekiri. 2002\.  What is the value of graphical displays in learning?  _Educational psychology review_ 14, 3 (2002), 261–312.
  * Viegas (2007) Fernanda B Viegas. 2007\.  The visual side of wikipedia. In _Proc. Hawaii International Conference on System Sciences (HICSS)_.
  * Wannemacher and Schulenburg (2010) Klaus Wannemacher and Frank Schulenburg. 2010.  Wikipedia in academic studies: corrupting or improving the quality of teaching and learning?  In _Looking toward the future of technology-enhanced education: Ubiquitous learning and the digital native_. IGI Global, 295–311.
  * Wikidata (2022) Wikidata. 2022\.  train (Q870).  https://www.wikidata.org/wiki/Q870
  * Wikidata (2022) Wikidata. 2022.  Wikiproject Cultural Heritage.  https://www.wikidata.org/wiki/Wikidata:WikiProject_Cultural_heritage
  * Wikimedia Commons (2022) Wikimedia Commons. 2022\.  Commons:Quality Images.  https://commons.wikimedia.org/wiki/Commons:Quality_images
  * Wikimedia Foundation, Inc. (2022a) Wikimedia Foundation, Inc. 2022a.  Add-an-Image Structured Task.  https://www.mediawiki.org/wiki/Growth/Personalized_first_day/Structured_tasks/Add_an_image
  * Wikimedia Foundation, Inc. (2022b) Wikimedia Foundation, Inc. 2022b.  Commons:Project scope.  https://commons.wikimedia.org/wiki/Commons:Project_scope
  * Wikimedia Foundation, Inc. (2022c) Wikimedia Foundation, Inc. 2022c.  Map of Visual Knowledge Gaps.  https://meta.wikimedia.org/wiki/Research:Map_of_Visual_Knowledge_Gaps
  * Wikimedia Foundation, Inc. (2022d) Wikimedia Foundation, Inc. 2022d.  Why We Read Wikipedia Data.  https://figshare.com/articles/dataset/Why_the_World_Reads_Wikipedia/7579937?file=17308100
  * Wikimedia Foundation, Inc. (2022e) Wikimedia Foundation, Inc. 2022e.  Wikimedia Foundation Mission.  https://wikimediafoundation.org/about/mission/
  * Wikimedia Foundation, Inc. (2022f) Wikimedia Foundation, Inc. 2022f.  Wikipedia:Barnstars.  https://en.wikipedia.org/wiki/Wikipedia:Barnstars
  * Wikimedia Foundation, Inc. (2022g) Wikimedia Foundation, Inc. 2022g.  Wikipedia:No original research.  https://en.wikipedia.org/wiki/Wikipedia:No_original_research
  * Wikipedia (2022a) English Wikipedia. 2022a.  Wikipedia Manual of Style.  https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Images
  * Wikipedia (2022b) English Wikipedia. 2022b.  Wikiproject Historic Areas.  https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Historic_sites
  * Wilkinson and Huberman (2007) Dennis M. Wilkinson and Bernardo A. Huberman. 2007.  Cooperation and Quality in Wikipedia. In _Proceedings of the 2007 International Symposium on Wikis_ (Montreal, Quebec, Canada) _(WikiSym ’07)_. Association for Computing Machinery, New York, NY, USA, 157–164.  https://doi.org/10.1145/1296951.1296968
  * Ye et al. (2017) Teng Ye, Katharina Reinecke, and Lionel P Robert Jr. 2017\.  Personalized feedback versus money: the effect on reliability of subjective data in online experimental platforms. In _Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing_. 343–346.