ʻ ‘

11institutetext: University of Hawaiʻi at Mānoa
Honolulu, Hawaiʻi, USA
11email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Impostor Syndrome in Final Year Computer Science Students: An Eye Tracking
and Biometrics Study

Alyssia Chen  0009-0002-5684-8601 Carol Wong  Katy Tarrit  Anthony Peruma
0000-0003-2585-657X

###### Abstract

Imposter syndrome is a psychological phenomenon that affects individuals who
doubt their skills and abilities, despite possessing the necessary
competencies. This can lead to a lack of confidence and poor performance.
While research has explored the impacts of imposter syndrome on students and
professionals in various fields, there is limited knowledge on how it affects
code comprehension in software engineering. In this exploratory study, we
investigate the prevalence of imposter syndrome among final-year undergraduate
computer science students and its effects on their code comprehension
cognition using an eye tracker and heart rate monitor. Key findings
demonstrate that students identifying as male exhibit lower imposter syndrome
levels when analyzing code, and higher imposter syndrome is associated with
increased time reviewing a code snippet and a lower likelihood of solving it
correctly. This study provides initial data on this topic and establishes a
foundation for further research to support student academic success and
improve developer productivity and mental well-being.

###### Keywords:

Cognitive Load and Performance Eye Tracking Biometrics Heart Rate Impostor
Syndrome Code Comprehension Program Comprehension Computer Science
Undergraduate Students

##  1 Introduction

Impostor syndrome, also known as impostor phenomenon, fraud syndrome,
perceived fraudulence, or impostor experience, is a psychological phenomenon
characterized by persistent self-doubt of intellect, skills, or
accomplishments and feelings of fraudulence or inadequacy despite evidence of
one’s competence and accomplishments [5, 6]. It is thought to affect high-
achieving individuals across various domains, including higher education and
high-skill professions [4, 19]. Although there have been studies examining how
women and minority groups cope with imposter syndrome [10], and how other
variables are correlated such as self-esteem [11], and self-efficacy [14],
there is limited research on the connection between impostor syndrome and
undergraduate computer science students.

As undergraduate students progress through their computer science degree, they
not only acquire new skills but also refine existing ones, ultimately becoming
proficient in designing and implementing software systems. By the time they
reach the end of their academic journey, these students possess the
qualifications necessary to begin their careers in the tech industry. It is
natural to assume that at this stage, these students are confident in their
skills and feel well-prepared for professional success. However, previous
research indicates that students enrolled in STEM programs, including those
related to computer science, often experience feelings of inadequacy,
potentially impacting their confidence in their skills [15]. As such, this
study aims to examine the prevalence and relationship of impostor syndrome
with code comprehension performance in fourth-year computer science students
through an augmented cognition approach.

###  1.1 Goal & Research Questions

Code comprehension is an essential activity in software development and
maintenance. Source code conveys the system’s behavior, through identifier
names [13], which developers rely on to understand the code they are working
on to fix defects or incorporate feature changes [21]. As computer science
students prepare to enter the workforce, feelings of self-doubt and insecurity
associated with impostor syndrome can undermine their ability to comprehend
codebases, thereby negatively impacting their confidence and job performance.
Therefore, this study aims to understand how impostor syndrome impacts code
comprehension. We envision our study laying the foundation for future research
in this area while also providing initial insights to improve computer science
education and support students as they transition to professional roles. To
this extent, we aim to answer the following research questions (RQs):

RQ 1: To what extent are final-year undergraduate computer science students
confident in their program comprehension skills? This research question
intends to examine the level of impostor syndrome exhibited by computer
science students who are about to graduate. The research question assesses the
demographic factors closely associated with impostor syndrome.

RQ 2: How does impostor syndrome affect cognitive processes involved in
comprehending code? This research question aims to better understand cognitive
processes involved in comprehending code, and how impostor syndrome can affect
these processes. This is achieved by tracking eye movements, monitoring heart
rates, and taking other measurements, such as the confidence of participants,
time taken to complete a code question, and how often participants answered
the coding question correctly.

###  1.2 Contribution

The main contributions from this work are as follows:

  * •

We provide preliminary yet promising findings on the extent to which impostor
syndrome affects the code comprehension cognition of experienced undergraduate
students.

  * •

This study establishes groundwork for further research exploring interventions
against impostor syndrome.

  * •

We make our dataset publicly accessible for replication/extension purposes.

###  1.3 Paper Structure

The rest of this paper is organized as follows: Section 2 presents a review of
related work on imposter syndrome in software engineering. Section 3 describes
the methodology used for this study. Section 4 provides answers to both
research questions and reports on the results of the proposed experiments.
Section 5 discusses the potential threats to the validity of the study.
Finally, Section 6 concludes by summarizing the findings and suggesting
potential directions for future work.

##  2 Related Work

In this section, we report on related work in this area. While there exist
studies that examine imposter syndrome in students and industry professionals,
we limit our focus to the literature that addresses imposter syndrome among
students who are either taking a computing course or studying a computing
subject/topic, as well as software engineering professionals.

In a survey with 200 undergraduate computer science students, Rosenstein et
al. [15] show that 57% of the surveyed students exhibited frequent feelings of
the Impostor Phenomenon. The authors also highlight that women experienced
these feelings more frequently (71%) compared to men (52%). Heels and Devlin
[8] examine the roles chosen by female students in a large software
engineering team project and report that despite strong academic backgrounds,
female students tend to opt for less technical roles than male students. The
authors recommend exploring how unconscious bias or imposter syndrome affects
female students. Interviews with 16 Digital Technologies teachers from primary
and secondary schools by Varoy et al. [20] show that female students
experience imposter syndrome more than male students and feel they are not
making as much progress as male students, even when they are doing better
work. The teachers note that male students tend to loudly proclaim their
achievements, while female students work more quietly and cooperatively. This
led the female students to underestimate their own abilities and progress. In
a study on coding bootcamps, Thayer and Ko [17] found that imposter syndrome
can act as an informal boundary to entering the software industry and persist
even after gaining experience and employment. Additionally, participants in
coding bootcamps can also experience imposter syndrome, similar to those in
the software industry.

A survey by Ginter [7] of 104 software engineers shows that individuals
exhibiting feelings of imposter syndrome usually had insecure attachment
styles, particularly anxiety and avoidance. The study also found that anxious
attachment styles in individuals with imposter syndrome were linked to higher
levels of depression and anxiety. In a study that surveyed 94 women employed
at a global technology company, Trinkenreich et al. [18] note that imposter
syndrome was a challenge women faced in software development teams and could
lead to them leaving the organization. In an online survey of 134 Finnish
women, Hyrynsalmi and Hyrynsalmi [9] reports that women are motivated to
transition to the software industry but encounter challenges such as a lack of
career counseling, uncertainty about the required education, and issues
related to self-esteem and imposter syndrome.

##  3 Study Design

In this section, we provide details about the design of this study. At a high
level, the study consisted of participants reviewing a set of code snippets
and answering questions through online questionnaires. While reading code
snippets, participants’ eye movements and heart rates were monitored. We will
elaborate on these activities below. The experiment took place in a private
room without any windows, where only the participant and one investigator
(i.e., an author) were present. A 24-inch monitor was used to display code
snippets and questionnaires. The investigator used a secondary monitor, mouse,
and keyboard to manage the experiment. After the calibration stage and before
starting the experiment, each participant went through a trial run. This
allowed to check if the eye tracker and biometric devices were functioning
adequately but also to make sure participants understood and got familiar with
the task. A couple trial code snippets were then displayed during this process
and have not been included in the results. This study was approved by the
Institutional Review Board of our institute and all participants provided
consent prior to participating. The high-resolution images of the code
snippets, survey questionnaire, and participant data are available at: [1].

###  3.1 Survey Design

As part of our study, we used three types of online questionnaires - a
demographic pre-questionnaire, a code snippet questionnaire, and an impostor
syndrome post-questionnaire. To construct and host these questionnaires, we
leveraged Qualtrics. All participants answered survey questions using the same
workstation that was connected to the eye tracker and biometrics device.
Below, we have provided a brief description of each questionnaire.

Prior to code comprehension activities and questions, participants completed a
pre-questionnaire capturing their demographics. All questions were required
and are shown in Table 1.

Table 1: Pre-Questionnaire capturing demographic details. No. |  Question |  Type
---|---|---
1 |  What best describes your gender? |  Single-Choice
2 |  What best describes your ethnicity? |  Multi-Choice
3 |  What is your current academic year? |  Single-Choice
4 |  What is your primary field of study/major? |  Single-Choice
5 |  Are you a first generation college student? |  Single-Choice
6 |  Do you have an immediate family member who has worked or is working in the software industry? (this includes internships) |  Single-Choice
7 |  How many years of experience in the software industry do you have? (this includes internships) |  Single-Choice
8 |  How many years of Java programming experience do you have? |  Single-Choice
9 |  What is your preferred programming language? |  Single-Choice
10 |  Do you have a software engineering-related job lined up post-graduation? |  Single-Choice
11 |  How would you rate your overall experience with your programming education? |  Single-Choice

During the coding analysis task, participants were presented with five
individual code snippets, each accompanied by one single-choice question. The
question required participants to either identify the correct output of the
code or identify the number of the line where the code was causing a logical,
runtime, or compile-time error. Participants only had five minutes to read
each code snippet and answer the associated question. To reduce the
possibility of participants guessing the answer, each question included an “I
don’t know” option. Section 4 provides screenshots of each code snippet, along
with their associated question and multiple options for answer. Right after
participants provided an answer to the question related to a specific code
snippet, they were asked to provide feedback on the confidence-level for the
answer they selected. After completing code snippets’ comprehension
activities, participants were asked to fill out a post-questionnaire to assess
the extent of their imposter syndrome. For this study, Clance IP Scale
questions [5] were customized to focus only on source code analysis and
troubleshooting 111From The Impostor Phenomenon: When Success Makes You Feel
Like A Fake (pp. 20-22), by P.R. Clance, 1985, Toronto: Bantam Books.
Copyright 1985 by Pauline Rose Clance. Reprinted by permission. Do not
reproduce without permission from Pauline Rose Clance,
<EMAIL_ADDRESS>Those questions can be found in Table 2. All
participants were required to answer all questions in this questionnaire.

Table 2: Post-Questionnaire containing the customized Clance IP Scale questions that focus on source code analysis and troubleshooting. No. |  Question |  Type
---|---|---
1 |  I have often succeeded in programming tasks, even though I was afraid that I would not do well before I started working on it. |  Single-Choice
2 |  I can give the impression that I’m more competent in my programming skills than I really am. |  Single-Choice
3 |  I tend to avoid programming tasks and have a sense of dread when others assess my programming work. |  Single-Choice
4 |  When people praise me for my code analysis and troubleshooting abilities, I’m afraid I won’t be able to live up to their expectations of me in the future. |  Single-Choice
5 |  I sometimes think my success in code analysis and troubleshooting is due to external factors (i.e., environmental or people) rather than due to my skills. |  Single-Choice
6 |  I’m afraid people important to me may find out that I’m not as capable at programming as they think I am. |  Single-Choice
7 |  I tend to remember the incidents in which I have not done my best in programming more than those times I have done my best. |  Single-Choice
8 |  I rarely do a programming task as well as I’d like to do it. |  Single-Choice
9 |  Sometimes I feel or believe that my success in analyzing and troubleshooting code has been the result of some kind of accident. |  Single-Choice
10 |  It’s hard for me to accept compliments or praise about my programming skills or accomplishments |  Single-Choice
11 |  At times, I feel my success in code analysis and troubleshooting has been due to some kind of luck. |  Single-Choice
12 |  I’m disappointed at times in my code analysis and troubleshooting accomplishments and think I should have accomplished much more. |  Single-Choice
13 |  Sometimes I’m afraid others will discover how much code analysis and troubleshooting knowledge or ability I really lack. |  Single-Choice
14 |  I’m often afraid that I may fail at a new code analysis and troubleshooting assignment or undertaking, even though I generally do well at what I attempt. |  Single-Choice
15 |  When I’ve succeeded at programming and received recognition for my accomplishments, I have doubts that I can keep repeating that success. |  Single-Choice
16 |  If I receive a great deal of praise and recognition for my code analysis and troubleshooting accomplishments, I tend to discount the importance of what I’ve done. |  Single-Choice
17 |  I often compare my code analysis and troubleshooting abilities to those around me and think they may be more intelligent than I am. |  Single-Choice
18 |  I often worry about not succeeding with a programming task, even though others around me have considerable confidence that I will do well. |  Single-Choice
19 |  If I’m going to gain recognition for my code analysis and troubleshooting skills, I hesitate to tell others until it is an accomplished fact. |  Single-Choice
20 |  I feel bad and discouraged if I’m not "the best" or at least "very special" in code analysis and troubleshooting situations that involve achievement. |  Single-Choice

###  3.2 Code Snippets

In this study, each participant was required to review five code snippets and
answer their associated question. Code snippets were in the Java programming
language as it is widely used in multiple courses at the University, and all
students are familiar with this programming language. Code snippets covered
concepts such as data structures, recursion, sorting, and string analysis,
which are taught in undergraduate-level courses and are usually asked during
job interviews in industry. Table 3 provides a summary of code snippets. The
code snippets and their associated question are shown in Section 4.

Table 3: Description of the five code snippets utilized in this study.

Id | Description | Task
---|---|---
1 |  This code snippet involves array size manipulation within a loop that will cause a runtime issue. Unlike the error in the other code snippets, participants were not explicitly informed that an error exists in the code. |  Determine the output
2 |  This is an implementation of the bubble sort algorithm. However, there is an error in the condition used in the loop, which will lead to the list being sorted incorrectly. The method’s name is called ‘bubbleSort’, so the participant is aware of the intended behavior of the code. |  Identify the number of the line where the error occurs
3 |  This is a recursive function that prints a sequence of digits. There are no errors in the code. |  Determine the output
4 |  This code snippet accepts a provided string and prints the length of the last word in the provided string. The names of the identifiers in the code do not indicate the code’s behavior. There are no errors in this code. |  Determine the output
5 |  This code checks if a given string is a palindrome. The implementation contains an error in the calculation that results in a runtime error. |  Identify the number of the line where the error occurs

###  3.3 Eye Tracker & Biometric Device

Eye movements and physiological samples such as heart rate were recorded
respectively using the Gazepoint GP3 HD Eye Tracker and the Biometrics device.
These devices are research-grade and are commonly used for academic as well as
medical research [2, 3]. The eye tracker has a sampling rate of 150 Hz to
reduce the chance of loss of tracking and a 0.5 – 1.0 degree of visual angle
accuracy. During the experiment, participants sat on a chair facing a computer
monitor where code snippets and surveys were presented to them. Prior to the
experiment, the camera was calibrated using the Gazepoint Control software.
Participants were asked to make saccades to nine targets presented on the
screen for calibration purposes. Those nine targets were presented at the top,
at the middle and at the bottom of the screen going from the left through the
middle to the right of the screen. The heart rate was measured using a finger
sensor module. The Gazepoint Analysis software was utilized to manage the
experiment, including starting/stopping the eye tracker and the creation of
areas of interest (i.e., AOIs).

###  3.4 Data Preprocessing

Once data collection was completed, data for analysis was prepared using
Python. The temporal gaze point data (a single gaze point represents where a
user looked and when in milliseconds) from Gazepoint was first cleaned by
removing durations of time where: (1) the code was not on the screen due to
latency from Qualtrics when displaying the survey question, (2) the time
between code snippets, and (3) the participant was looking away from the
screen and was not focused on the code snippet. The times for (1) and (3) were
determined manually by two investigators watching the video recordings with
precision up to three decimal places. The cutoff time for (2) was automated by
trimming the data up to when the participant submitted their answer for a code
snippet question.

The following aggregate measures were calculated for each participant: average
heart rate, total time spent on a snippet, and most looked at AOI
(specifically, only AOIs containing code were analyzed for this study; answer
choices below each code snippet were not considered). Duration spent in each
AOI was determined by summing the differences in time between the first
instance when a gaze point was identified to be within an AOI and the first
instance when the participant moved to a different AOI. All durations for the
AOIs were then normalized by the number of words, excluding non-alphanumeric
characters (as described in Sharafi et al.[16]).

Other calculations for the data included determining levels of imposter
syndrome (detailed in RQ1 of Section 4) and determining the correctness of
answers (a binary true/false if the answer was correct) for each code snippet
which were determined by two investigators. Additionally, recoding of ordinal
variables to be numerical were coded starting from the number zero and non-
codeable responses were designated as NaN.

###  3.5 Participants

In this study, fifteen final-year undergraduate students enrolled in the
Computer Science program at the Information and Computer Sciences Department
of the University of Hawaiʻi at Mānoa were recruited. These participants were
not given any monetary compensation or extra credit for participating in the
study; participation was voluntary.

Out of the fifteen participants who took the survey, nine of them identified
as male, five as female, and one as non-binary/third gender. Moreover among
the participant pool, ten identified themselves as Asian only, one as
White/Caucasian, one as Black/African American, and the remaining three
participants identified with two or more ethnicities.

In terms of Java programming experience, eight participants indicated they had
one to two years of experience, four participants mentioned they had three to
five years of experience, and three participants reported they had less than
one year of experience. Additionally, six participants had between one to two
years of industry experience, five participants had less than a year of
industry experience, two participants had three to five years of industry
experience, and the remaining two participants had no prior industry
experience. Table 4 provides a breakdown of participants’ demographics.

Table 4: Demographic details of the participants in the study.

| Participant
---
ID
Gender | Ethnicity |  | Years of Exp.
---
in Software
Industry
| Years of Exp.
---
in Java
1 | Male | White or Caucasian | 1-2 years | 1-2 years
2 | Female | Asian | 1-2 years | 1-2 years
3 | Female | Asian, White or Caucasian | 1-2 years | 3-5 years
4 | Male | Asian | None | 3-5 years
5 | Male | Asian | >1 year | 3-5 years
6 | Female | Asian | 1-2 years | 1-2 years
7 | Female | Asian | 3-5 years | 3-5 years
8 |  | Non-binary /
---
third gender
Asian | >1 year | >1 year
9 | Male | Black or African American | None | 1-2 years
10 | Male | Asian | 1-2 years | >1 year
11 | Female | Asian, Native Hawaiian | 1-2 years | 1-2 years
12 | Male | Asian | >1 year | 1-2 years
13 | Male | Asian | 3-5 years | 1-2 years
14 | Male | Asian | >1 year | >1 year
15 | Male |  | American Indian or Alaska Native,
---
Asian, White or Caucasian
>1 year | 1-2 years

###  3.6 Pilot Study

We followed the best practice of conducting a pilot study to identify any
flaws in our methodology. For this purpose, we recruited four students to
participate in a pilot study, who were not included in the actual study. All
data collected during the pilot study were discarded after its completion.
During the pilot run, we identified certain survey questions that needed to be
reworded to improve clarity as well as code snippets that required consistent
formatting. We also made improvements to the positioning of the eye tracker
and the participant’s chair. Furthermore, the pilot study helped us construct
a script/instructions to follow when conducting the actual study.

##  4 Results

In this section, we report the study’s results by answering our RQs.

###  4.1 RQ 1: To what extent are final-year undergraduate computer science
students confident in their program comprehension skills?

Table 5: Range of values associated with each IPC level IPC Level | Scores
---|---
Few | <40
Moderate | 41 - 60
Frequent | 61 - 80
Intense | >80

Based on the Clance Imposter Phenomenon (IP) Scale [5], participants were
sorted into four Imposter Phenomenon Characteristics (IPC) categories: Few IPC
for those who demonstrated few imposter syndrome characteristics, Moderate IPC
for those with moderate characteristics, Frequent IPC, and Intense IPC, with
each corresponding to the severity level of IP. These levels correspond to how
frequently and seriously IP interferes with a person’s life.

Once participants filled out the post-questionnaire based on a version of the
Clance IP Scale adapted to the task, their score was calculated as follows:
Each question had a range of multiple-choice answers, which were "not at all
true," "rarely," "sometimes," "often," and "very true." Each answer mapped to
a numerical value from 1 to 5, respectively. These values are then tallied to
create an aggregate score for each participant and used to classify each
participant into one of the four IPC groups. The range of values associated
with each IPC category can be found in Table 5.

Of the fifteen participants, three had Few IPC, five had Moderate IPC, five
had Frequent IPC, and two had Intense IPC. Of the three participants with Few
IPCs, all three identified themselves as male. In the Moderate and Frequent
IPC groups, each comprising five individuals, two participants identified
themselves as female and three as male. Amongst the two participants with
Intense IPC, one identified as female and the other one as non-binary/third
gender.

A t-test to compare the IPC scores between males and females (the participant
who responded "non-binary" was omitted) found that the nine males (M = 50.56,
SD = 12.32) compared to the five females (M = 68.80, SD = 12.28) scored
significantly lower on the modified Clance IP scale, t(12) = -2.658, p = .021.
Therefore, females were associated with higher levels of the imposter
phenomenon, particularly in relation to source code analysis and
troubleshooting.

One-way ANOVA tests were additionally conducted to compare how IPC scores were
related to software industry experience and Java experience. Findings show
that there is no significant difference in IPC scores so there is no
relationship between these variables (p > .05 for both).

Summary for RQ1. We found that female students had higher characteristics of
imposter syndrome. The number of years of experience in the software industry
or the tested programming language did not seem to correlate to the level of
characteristics of imposter syndrome.

###  4.2 RQ 2: How does imposter syndrome affect the cognitive processes
involved in comprehending code?

To answer this research question, we examine the eye tracking and biometrics
results of participants comprehending each code snippet in the study. Below,
for each code snippet, we discuss the average heart rate, Areas of Interest
(AOI) metrics, time spent by participants on the snippet, the percentage of
correct answers, and the extent to which participants are confident with their
answers. Additionally, Figure 1 displays all five code snippets that were
included in our study, along with their corresponding question.

AOIs are regions of a stimulus from which quantitative metrics can be derived
[16]. Each AOI contains its own set of metrics, e.g., the average time
participants took to look at a specific AOI. Each line of code, excluding
those solely comprising of curly braces, was designated an AOI. Each stimulus
had an average of 13.4 AOIs. As mentioned in Section 3, each AOI was
normalized by the number of words it contained on its line in order to allow
fair comparison between AOIs (words were characterized as sequences of
alphanumeric characters that are separated by spaces after excluding non-
alphanumeric characters).

The AOIs were then grouped into one of ten different categories, depending on
the functionality of the line of code. These categories are as follows: else
statement, for loop, if statement, import statement, method call, method
declaration, recursive method call, return statement, variable assignment, and
variable declaration. An example of a method declaration is “public static
void main(String[] args) {”, while an example of a method call is
“list.add(1).”

Figure 1: The five code snippets that were part of our study along with their
associated question and answer choices.

Code Snippet 1 - Array Size Manipulation

(a) Average Heart Rate

(b) Time Spent on Snippet

(c) % Correctly Answered

Figure 2: Biometrics and Other Measures Averaged per IPC Group for Snippet 1

Average heart rate and time spent was further averaged within each IPC
grouping (Few, Moderate, Frequent, and Intense IPC). The percentage of
participants who scored correctly in each IPC grouping is displayed in the
last figure.

It was found that for Code Snippet 1, only one participant from the Moderate
IPC group answered the problem correctly (see Figure 2(c)). The lowest median
for the average heart rate occurred in the Few IPC group, whereas the highest
median average heart rate occurred in the Moderate IPC group (see Figure
2(a)). The amount of time participants took before submitting an answer was
the lowest for those in the Few IPC group and the highest in the Intense IPC
group (see Figure 2(b)).

The lines of code that participants spent the longest duration on were import
statements for Few IPC, Moderate IPC, and Frequent IPC (for each row in Table
6, the percentage was highest for these three IPC groups). Those in the
Intense IPC spent the longest duration on if statements (23.87%). For all but
those in Frequent IPC, the least amount of time was spent on method
declarations.

Table 6: Duration spent on a Code Category in Snippet 1

|  For Loop |  If Stmt |  Import Stmt |  Method Call |  Method Decl |  Var Decl |  Multiple
---|---|---|---|---|---|---|---
Few IPC |  16.18% (0.46) |  18.28% (0.52) |  21.93% (0.62) |  21.29% (0.6) |  10.71% (0.3) |  11.14% (0.32) |  0.47% (0.01)
Moderate IPC |  17.17% (0.8) |  12.7% (0.59) |  25.17% (1.17) |  18.25% (0.85) |  11.62% (0.54) |  14.51% (0.67) |  0.58% (0.03)
Frequent IPC |  17.76% (0.91) |  10.93% (0.56) |  30.31% (1.56) |  16.01% (0.82) |  12.66% (0.65) |  12.07% (0.62) |  0.24% (0.01)
Intense IPC |  20.63% (1.18) |  23.87% (1.36) |  14.71% (0.84) |  17.65% (1.01) |  9.67% (0.55) |  11.26% (0.64) |  2.21% (0.13)

Row percentages for each IPC are shown in each cell with red highlights
denoting largest percentage and green highlights denoting smallest (ignoring
the Multiple category). The duration normalized by number of words are in
parenthesis.

Code Snippet 2 - Bubble Sort

(a) Average Heart Rate

(b) Time Spent on Snippet

(c) % Correctly Answered

Figure 3: Biometrics and Other Measures Averaged per IPC Group for Snippet 2

Average heart rate and time spent was further averaged within each IPC
grouping (Few, Moderate, Frequent, and Intense IPC). The percentage of
participants who scored correctly in each IPC grouping is displayed in the
last figure.

As shown in Figure 3, only two participants from the Frequent IPC group
answered the problem correctly. The average heart rate was the lowest in the
Moderate IPC group and Highest in the intense IPC group. Participants in the
Few IPC group spent the least amount of time and those in the Intense IPC
group spent the most.

Referring to Table 7, those from the Few IPC, Moderate IPC, and Frequent IPC
spent the least amount of time on the if statements, while those in the
Intense IPC group spent the least amount of time on variable assignments.
Besides the Moderate IPC group, every group spent the most time on method
declarations.

Table 7: Duration spent on a Code Category in Snippet 2

|  For Loop |  If Stmt |  Method Decl |  Var Asgmt |  Var Decl
---|---|---|---|---|---
Few IPC |  14.09% (0.66) |  8.33% (0.39) |  39.24% (1.84) |  11.82% (0.55) |  26.52% (1.24)
Moderate IPC |  16.45% (1.01) |  13.05% (0.8) |  24.17% (1.48) |  25.87% (1.59) |  20.46% (1.26)
Frequent IPC |  19.54% (1.69) |  10.92% (0.95) |  36.96% (3.2) |  12.79% (1.1) |  19.89% (1.72)
Intense IPC |  17.95% (1.01) |  13.02% (0.73) |  36.51% (2.06) |  12.7% (0.72) |  19.74% (1.11)

Code Snippet 3 - Recursion

(a) Average Heart Rate

(b) Time Spent on Snippet

(c) % Correctly Answered

Figure 4: Biometrics and Other Measures Averaged per IPC Group for Snippet 3

Average heart rate and time spent was further averaged within each IPC
grouping (Few, Moderate, Frequent, and Intense IPC). The percentage of
participants who scored correctly in each IPC grouping is displayed in the
last figure.

For Code Snippet 3, all of the participants in the Few IPC group answered the
problem correctly, whereas no participants from the Intense IPC group answered
correctly (Figure 4). One participant from the Moderate and Frequent IPC
groups answered correctly. The lowest median average heart rate was from the
Frequent IPC group, and the highest median was from the Intense IPC group.
Participants from the Intense IPC group spent the longest amount of time on
the snippet compared to those in the Few IPC group, who spent the shortest.

This snippet had less consistent results across the IPC groupings as can be
seen in Table 8. However, it can seen that those in the Few IPC group spent
the most time on if-statements and the least amount of time on recursive
calls, whereas those in the Intense IPC group spent most of their time on
method declarations and the least amount of time on if-statements.

Table 8: Duration spent on a Code Category in Snippet 3

|  If Stmt |  Method Call |  Method Decl |  Recursive Call
---|---|---|---|---
Few IPC |  31.25% (1.08) |  22.73% (0.79) |  30.88% (1.07) |  15.14% (0.52)
Moderate IPC |  28.7% (2.39) |  15.21% (1.27) |  14.06% (1.17) |  42.03% (3.5)
Frequent IPC |  24.38% (2.13) |  33.73% (2.95) |  21.83% (1.91) |  20.06% (1.75)
Intense IPC |  8.06% (0.6) |  34.43% (2.58) |  34.54% (2.59) |  22.96% (1.72)

Code Snippet 4 - String Analysis

(a) Average Heart Rate

(b) Time Spent on Snippet

(c) % Correctly Answered

Figure 5: Biometrics and Other Measures Averaged per IPC Group for Snippet 4

Average heart rate and time spent was further averaged within each IPC
grouping (Few, Moderate, Frequent, and Intense IPC). The percentage of
participants who scored correctly in each IPC grouping is displayed in the
last figure.

Referring to Figure 5, seven participants answered this question correctly,
including all three from Few IPC, three between Moderate and Frequent IPC, and
one from Intense IPC. The median average heart rate was similar between
Moderate, Frequent, and Intense IPC, but those in the Few IPC group had the
lowest average heart rate. For time spent on the snippet, participants with
Intense IPC had the longest average duration; participants with Few IPC had
the shortest average duration.

As seen in Table 9, participants with Few, Moderate, and Frequent IPC spent
the longest duration on variable assignments. For code categories with the
shortest duration, there was no consistency among the four IPC groups.

Table 9: Duration spent on a Code Category in Snippet 4

|  Else Stmt |  For Loop |  If Stmt |  Method Call |  Method Decl |  Return Stmt |  Var Asgmt |  Var Decl |  Multiple
---|---|---|---|---|---|---|---|---|---
Few IPC |  3.16% (0.26) |  7.95% (0.65) |  8.74% (0.72) |  9.61% (0.79) |  19.49% (1.6) |  17.82% (1.46) |  23.97% (1.97) |  7.7% (0.63) |  1.56% (0.13)
Moderate IPC |  11.28% (1.83) |  10.19% (1.66) |  9.96% (1.62) |  11.15% (1.81) |  11.59% (1.88) |  9.78% (1.59) |  20.56% (3.34) |  9.7% (1.58) |  5.76% (0.94)
Frequent IPC |  14.72% (2.29) |  4.6% (0.72) |  9.42% (1.47) |  9% (1.4) |  12.65% (1.97) |  10.71% (1.67) |  25.78% (4.02) |  10.58% (1.65) |  2.54% (0.4)
Intense IPC |  15.12% (1.52) |  7.69% (0.78) |  5.32% (0.54) |  7.72% (0.78) |  24.46% (2.47) |  8.23% (0.83) |  12.54% (1.26) |  15.35% (1.55) |  3.57% (0.36)

Code Snippet 5 - Palindrome

(a) Average Heart Rate

(b) Time Spent on Snippet

(c) % Correctly Answered

Figure 6: Biometrics and Other Measures Averaged per IPC Group for Snippet 5

Average heart rate and time spent was further averaged within each IPC
grouping (Few, Moderate, Frequent, and Intense IPC). The percentage of
participants who scored correctly in each IPC grouping is displayed in the
last figure.

For Code Snippet 5, eight participants answered correctly, again including all
three from Few IPC, one from Moderate IPC, and four from Frequent IPC (See
Figure 6). No participant from the intense IPC group answered correctly. Those
in the few IPC group had the lowest average heart rate, whereas those in the
intense IPC had the highest. Similar to the other snippets, participants with
Few IPC spent the least amount of time on the snippet, whereas those with
Intense IPC spent the most amount of time.

Referring to Table 10 all four IPC groupings spent the most time looking at
method declarations. Those with Few and Moderate IPC spent the least amount of
time on return statements, whereas those with Frequent and Intense IPC spent
the least amount of time on if statements.

Table 10: Duration spent on a Code Category in Snippet 5

|  Else Stmt |  For Loop |  If Stmt |  Method Decl |  Return Stmt |  Var Asgmt |  Var Decl
---|---|---|---|---|---|---|---
Few IPC |  5.5% (0.33) |  12.38% (0.75) |  6.02% (0.37) |  44.68% (2.72) |  5.4% (0.33) |  7.73% (0.47) |  18.28% (1.11)
Moderate IPC |  14.16% (1.81) |  10.32% (1.32) |  11.6% (1.48) |  25.57% (3.26) |  9.03% (1.15) |  14.81% (1.89) |  14.51% (1.85)
Frequent IPC |  10.19% (0.74) |  6.34% (0.46) |  6.34% (0.46) |  37.46% (2.74) |  7.27% (0.53) |  8.99% (0.66) |  23.41% (1.71)
Intense IPC |  20.88% (2.89) |  3.94% (0.54) |  1.67% (0.23) |  40.36% (5.58) |  9.97% (1.38) |  7.14% (0.99) |  16.04% (2.22)

####  4.2.1 Overall Analysis.

When averaging time across all 5 code snippets and for each imposter syndrome
level, there appears to be a trend where the median of the time spent
increases as participants score higher on Clance IP Scale (Figure 7(b)). When
averaging how many times participants in each IPC group answered questions
correctly across all code snippets, there was a downward trend (Figure 7(c)).
To measure the extent of the relationship between these variables, we
conducted a Pearson correlation test. The Pearson correlation test yielded
statistically significant (i.e., p-value < 0.05) correlation coefficients of
0.52, equating to a moderate positive correlation between IPC score and time
spent on a snippet and -0.52, equating to a moderate negative correlation
between IPC score and average correctness. While in Figure 7(a), there appears
to be a positive trend between average heart rate and IPC, it was not
statistically significant. Similarly, while there seems to be a pattern of
those with high IPC being less confident in their answers in 7(d), no
significant results were found.

(a) $AvgHeartRate$

(b) $AvgTimeSpent$

(c) $CorrectlyAnswered$

(d) $ConfidenceinAllSnippets$

Figure 7: Combined Results

To gain a deeper insight into overall how participants of varying IPC scores
read code, a Pearson Correlation 2-tailed test was conducted to compare
average normalized durations for each AOI code category (e.g., else-statement,
for-loop). The test yielded a statistically significant (i.e., p-value < 0.05)
correlation coefficient: 0.56 for the IPC score and Method Calls, equating to
a positive moderate correlation, and 0.52 for the IPC score and Method
Declarations. This could indicate that those with higher IPC scores tend to
spend more time looking at code associated with these two categories.

Summary for RQ2. Students with higher characteristics of imposter syndrome
were associated with spending longer on snippets and were less likely to
answer the question related to each code snippet correctly. Across all code
snippets, those with higher characteristics of imposter syndrome were more
likely to look at the Method Call and Method Declaration code categories.

##  5 Threats To Validity

Although our study sample size is small and limited to a single institute, it
remains valuable as an initial exploratory study providing a foundation for
further research. Furthermore, research shows that program comprehension
studies using an eye tracker have an average of 18.08 participants with a
standard deviation of 9.90 [12]. Likewise, due to the small sample size, the
small gender and race/ethnicity distribution in our participants’ pool may not
be representative of the general population. The number and type of code
snippets used in this study in order to evaluate comprehension threaten the
study’s validity, as they may not represent real-world codebases that students
will encounter in their careers. However, these code snippets represent basic
programming concepts that students learn when preparing for their degree and
are commonly asked in entry-level software engineering industry interviews.

The validity of our study may also be threatened by the limitations of the
Gazepoint GP3 HD Eye Tracker and Biometrics Kit. However, these are research-
grade devices that have been previously utilized in scientific publications
[2, 3]. Overall, while our study has some inherent limitations as an initial
investigation, it establishes a valuable foundation for further research on
this topic. Finally, as the experiment was conducted in a controlled lab
environment, it may not fully reflect real-world software development
situations and, therefore, could impact the validity of our results. However,
the controlled setting helped us focus on the effects of imposter syndrome on
code comprehension. Moreover, we followed established guidelines to set up our
eye-tracking experiment [16].

##  6 Conclusion & Future Work

Imposter syndrome is a psychological barrier that can negatively affect the
performance of students and professionals. While research on imposter syndrome
in software engineering does exist, little is known about how it affects code
comprehension cognition. In this exploratory study, we examine the level of
imposter syndrome final-year undergraduate computer science students exhibit
when comprehending code. We further measure the cognitive impact using an eye
tracker and heart rate monitor. Our findings show that: (1) students
identifying as males show lower levels of imposter syndrome, (2) higher levels
of imposter syndrome are associated with increased duration of time spent on a
snippet and lower chances of solving the problem correctly, and (3) those with
higher imposter syndrome levels are more likely to look at method declarations
and method calls. While limited in scope, our study establishes a foundation
for further research on imposter syndrome and its effects on core software
engineering competencies.

Our future work involves working with industry professionals in a similar
study to understand how imposter syndrome affects computer science
professionals in their day-to-day work, including its impact on mental health.

##  7 Acknowledgments

We thank students who took the time to participate in our study.

##  8 Data Availability Statements

The high-resolution images of the code snippets, survey questionnaires, and
participant data are available at: [1].

## References

  * [1] https://zenodo.org/doi/10.5281/zenodo.10656486
  * [2] Eye-tracking technology for academic research | gazepoint. https://www.gazept.com/academic-research/?v=7516fd43adaa
  * [3] Publications & citations | gazepoint. https://www.gazept.com/meet-the-team/publications/?v=7516fd43adaa
  * [4] Chakraverty, D.: Faculty experiences of the impostor phenomenon in STEM fields. CBE Life Sci Educ 21(4), ar84 (Dec 2022)
  * [5] Clance, P.R.: Clance impostor phenomenon scale. Personality and Individual Differences (1985)
  * [6] Clance, Pauline Rose Imes, S.A.: The imposter phenomenon in high achieving women: Dynamics and therapeutic intervention. Psychotherapy: Theory, Research & Practice 15(3) (1978). https://doi.org/10.1037/h0086006, https://doi.org/10.1037/h0086006
  * [7] Ginter, A.N.: Imposter Phenomenon, Insecure Attachment Style, and Mental Health in Software Engineers: An Examination of the Moderating Effect of Self Esteem. Ph.D. thesis, Alliant International University (2023)
  * [8] Heels, L., Devlin, M.: Investigating the role choice of female students in a software engineering team project. In: Proceedings of the 3rd Conference on Computing Education Practice. CEP ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3294016.3294028
  * [9] Hyrynsalmi, S., Hyrynsalmi, S.: What motivates adult age women to make a career change to the software industry? In: 2019 IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC). pp. 1–8 (2019). https://doi.org/10.1109/ICE.2019.8792630
  * [10] Maji, S.: “They Overestimate Me All the Time:” Exploring Imposter Phenomenon among Indian Female Software Engineers. Metamorphosis: A Journal of Management Research 20(2), 55–64 (Dec 2021). https://doi.org/10.1177/09726225211033699, http://journals.sagepub.com/doi/10.1177/09726225211033699
  * [11] Naser, M.J., Hasan, N.E., Zainaldeen, M.H., Zaidi, A., Mohamed, Y.M.A.M.H., Fredericks, S.: Impostor Phenomenon and Its Relationship to Self-Esteem Among Students at an International Medical College in the Middle East: A Cross Sectional Study. Frontiers in Medicine 9, 850434 (Apr 2022). https://doi.org/10.3389/fmed.2022.850434, https://www.frontiersin.org/articles/10.3389/fmed.2022.850434/full
  * [12] Obaidellah, U., Al Haek, M., Cheng, P.C.H.: A survey on the usage of eye-tracking in computer programming. ACM Comput. Surv. 51(1) (jan 2018). https://doi.org/10.1145/3145904
  * [13] Peruma, A., Mkaouer, M.W., Decker, M.J., Newman, C.D.: An empirical investigation of how and why developers rename identifiers. In: Proceedings of the 2nd International Workshop on Refactoring. p. 26–33. IWoR 2018, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3242163.3242169, https://doi.org/10.1145/3242163.3242169
  * [14] Pákozdy, C., Askew, J., Dyer, J., Gately, P., Martin, L., Mavor, K.I., Brown, G.R.: The imposter phenomenon and its relationship with self-efficacy, perfectionism and happiness in university students. Current Psychology (May 2023). https://doi.org/10.1007/s12144-023-04672-4, https://doi.org/10.1007/s12144-023-04672-4
  * [15] Rosenstein, A., Raghu, A., Porter, L.: Identifying the prevalence of the impostor phenomenon among computer science students. In: Proceedings of the 51st ACM Technical Symposium on Computer Science Education. p. 30–36. SIGCSE ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3328778.3366815
  * [16] Sharafi, Z., Sharif, B., Guéhéneuc, Y.G., Begel, A., Bednarik, R., Crosby, M.: A practical guide on conducting eye tracking studies in software engineering. Empirical Software Engineering 25, 3128–3174 (2020)
  * [17] Thayer, K., Ko, A.J.: Barriers faced by coding bootcamp students. In: Proceedings of the 2017 ACM Conference on International Computing Education Research. p. 245–253. ICER ’17, Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3105726.3106176
  * [18] Trinkenreich, B., Britto, R., Gerosa, M.A., Steinmacher, I.: An empirical investigation on the challenges faced by women in the software industry: a case study. In: Proceedings of the 2022 ACM/IEEE 44th International Conference on Software Engineering: Software Engineering in Society. p. 24–35. ICSE-SEIS ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3510458.3513018
  * [19] Trinkenreich, B., Wiese, I., Sarma, A., Gerosa, M., Steinmacher, I.: Women’s participation in open source software: A survey of the literature. ACM Trans. Softw. Eng. Methodol. 31(4) (aug 2022). https://doi.org/10.1145/3510460
  * [20] Varoy, E., Luxton-Reilly, A., Lee, K., Giacaman, N.: Understanding the gender gap in digital technologies education. In: Proceedings of the 25th Australasian Computing Education Conference. p. 69–76. ACE ’23, Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3576123.3576131
  * [21] Von Mayrhauser, A., Vans, A.M.: Program comprehension during software maintenance and evolution. Computer 28(8), 44–55 (1995). https://doi.org/10.1109/2.402076


# Chat BCG: Can AI Read Your Slide Deck?

Nikita Singh
<EMAIL_ADDRESS>Rob Balian
Reprompt AI
<EMAIL_ADDRESS>Lukas Martinelli
Reprompt AI
<EMAIL_ADDRESS>
###### Abstract

Multimodal models like GPT4o and Gemini Flash are exceptional at inference and
summarization tasks, which approach human-level in performance. However, we
find that these models underperform compared to humans when asked to do very
specific ’reading and estimation’ tasks, particularly in the context of visual
charts in business decks. This paper evaluates the accuracy of GPT 4o and
Gemini Flash-1.5 in answering straightforward questions about data on labeled
charts (where data is clearly annotated on the graphs), and unlabeled charts
(where data is not clearly annotated and has to be inferred from the X and Y
axis).
On labeled charts, we found that both GPT 4o and Gemini Flash are consistently
inaccurate on specific types of charts (multiple figures in one chart, stacked
charts, waterfall charts). Human error rates are estimated to be under 5%,
while GPT-4o and Gemini Flash have error rates of 16% and 14% 111Measured as %
of data points perfectly matching source of truth value., respectively. Both
LLMs often make similar mistakes, such as consistently misreading a ’3’ as an
’8’ or mislabelling a negative number as positive. Across charts, neither
model consistently outperforms the other.
Similiarly, on unlabeled charts, both GPT and Gemini have similiar failure
points. The models were tasked with estimating a number from an unlabeled
chart by reading the X and Y axis. As expected, perfectly matched answers were
low, with error rates as high as 79% for Gemini and 83% for GPT 4o. Here, we
measured the magnitude of the error for each model, compared to a (human read)
source of truth value. The errors were substantial, with average deviations of
53% for Gemini Flash-1.5 and 55% for GPT-4o 222Measured as mean absolute
percentage difference (see methodology for formula details)., compared to a
human margin of error estimated at 10-20%. Driving the deviations are
consistently small errors in estimation, but also bigger deviations driven by
models misreading labels/numbers. For instance, GPT 4o in one instance misread
2015 as 2009 and estimated the number for the wrong year.
We conclude that these models aren’t currently capable of reading a deck
accurately end-to-end if it contains any complex or unlabeled charts. Even if
a user created a deck of only labeled charts, the model would only be able to
read 7-8 out of 15 labeled charts perfectly end-to-end.

##  1 Introduction

With the advanced vision capabilities of GPT-4o and Gemini Flash, an important
question arises regarding the accuracy of these functionalities in practical
business applications. Our assumption was that multimodal models are good at
reading and summarizing charts. When given an image of a slide deck, they do a
good job of summarizing key insights from it, often including relevant data
points.
Existing research into this question has evaluated the efficacy of LLM’s when
parsing tables [3], concluding that the LLMs were highly sensitive to input
prompts which drive performance. Other works also evaluate LLMs ability to
reason and read mathematical graphs [2] and find that GPT models outperform
alternatives.
This paper aims to explore whether multimodal models perform well on a variant
of this skill - answering straightforward questions that require the models to
pick out a number from a slide deck. We test this by asking the models
questions about data directly printed on charts (if labeled) or asking the
model to estimate data points from the chart (if unlabeled). We ensure that
the model does not need to perform any mathematical calculations. We then
measure the accuracy of responses across different types of charts to answer
the following specific questions:

  * •

How accurately can multimodal models with advanced vision capabilities read
data from labeled charts? Is there a consistent accuracy advantage for one
model over the other?

  * •

How accurately can the models estimate numerical data from unlabeled charts?
On average, how ’incorrect’ is their estimation?

In the next section, we dive into error rates across different chart types,
broken into two categories - labeled and unlabeled.

##  2 How accurate are multimodal models in reading charts?

###  2.1 Methodology

We sampled 31 charts and classified them into 2 broad categories:

  * •

Labeled (15 charts): This category includes simple bar charts, line charts,
and multi-bar charts where each data point is explicitly printed on the chart.
We also included some more complex labeled charts, such as waterfall charts,
stacked charts, bubble charts, and connected bar charts.

  * •

Unlabeled (16 charts): This category is primarily focused on charts that do
not have explicitly printed data points and require some estimation of data by
’reading the position’ relative to the X and Y axis. It does not include
charts that do not have any scale at all. The dataset includes simpler
unlabeled charts such as bar charts, line charts, mixed charts, and bubble
charts, as well as some more complex unlabeled charts such as moon charts and
dot charts.

For each chart, we created a dataset of questions. These questions are
restricted to 3 types:

  * •

Identify a specific data point

  * •

Identify the largest/lowest data point

  * •

Count the number of data points

The goal of these questions is to test the ability to read and interpret data
directly from the chart, without requiring any intensive computation.

We then evaluate the responses of both the models on two metrics:

  * •

Match Rate %: This metric is used for labeled charts where the model is
reading data printed on the chart, but reported for both. It was calculated as
-

$MatchRate=\left(\frac{\text{Number of perfectly matched answers}}{\text{Total
number of questions}}\right)\times 100$

  * •

Mean Absolute Error (MAE): This metric is reported for unlabeled charts where
the model is ’estimating’ the number from the graph. It measures the average
magnitude of the errors in a set of predictions, without considering their
direction. This was calculated as -

$MAE=\frac{1}{n}\sum_{i=1}^{n}\left|A_{i}-P_{i}\right|$

where:

    * –

$n$ is the number of data points,

    * –

$A_{i}$ is the actual value for the $i$-th data point,

    * –

$P_{i}$ is the predicted value for the $i$-th data point.

  * •

Mean Absolute Percentage Error (MAPE): This metric is reported for unlabeled
charts where the model is ’estimating’ the number from the graph. It can be
read as how divergent the model’s answer was from the correct answer,
normalized to a percentage. This was calculated as -

$MAPE=\frac{1}{n}\sum_{i=1}^{n}\left|\frac{A_{i}-P_{i}}{A_{i}}\right|\times
100$

where:

    * –

$n$ is the number of data points,

    * –

$A_{i}$ is the actual value for the $i$-th data point,

    * –

$P_{i}$ is the predicted value for the $i$-th data point.

These metrics are then calculated at an aggregated level across all the charts
and reported for labeled and unlabeled datasets separately.

####  2.1.1 Methodology Walk-Through for Labeled Chart

For instance, in the labeled chart in Figure 1 (below), we generated questions
and measured the match % as given in Table 1

Figure 1: Example of labeled Chart Table 1: Comparison of GPT and Gemini answers for labeled chart Question | Correct Answer | GPT Answer | Gemini Answer
---|---|---|---
what % of the employee workforce is white in FY23? | 55 | 55 | 55
what % of the employee workforce is female in FY21? | 49 | 49 | 51
what % of the employee workforce is ’two or more races’ in FY22? | 3 | 3 | 8
what % of the employee workforce elected not to provide their race in FY21? | 4 | 4 | 4
what % of the employee workforce is Asian in FY23? | 22 | 22 | 7
Match % |  | 100% | 60%

####  2.1.2 Methodology Walk-Through for Unlabeled Chart

For unlabeled charts (see Figure 2 below), we generated questions and
estimated the MAPE, as given in Table 2

Figure 2: Example of Unlabeled Chart Table 2: Comparison of GPT and Gemini answers for unlabeled chart Question | Correct Answer | GPT Answer | Gemini Answer
---|---|---|---
How many deals happened in 2018? | 30 | 20 | 10
How much funding (in millions) was deployed in 2021? | 1500 | 1500 | 1500
What was the deal count in 2019? | 40 | 20 | 40
How much funding (in millions) was deployed in 2017? | 100 | 0.5 | 10
What was the highest deal count between 2017 and 2022? | 110 | 120 | 110
What was the highest funding (in millions) between 2017 and 2020? | 1450 | 1000 | 300
MAPE % |  | 37% | 39%

###  2.2 Summary Findings

Using the above methodology, the following table reports the overall match %
and MAPE % for labeled and unlabeled charts across the sample of 31 charts:

Table 3: Perfect Match Percentage for Labeled Charts Chart Type | GPT Match % | Gemini Match %
---|---|---
Labeled Charts | 84% | 86%
Table 4: MAPE Percentage for Unlabeled Charts Chart Type | GPT MAPE % | Gemini MAPE %
---|---|---
Unlabeled Charts | 55% | 53%

###  2.3 Labeled Charts: What is the match rate across charts?

Across 15 charts and a total of 71 questions, GPT 4o reads 8 charts perfectly
end-to-end and Gemini Flash-1.5 read 7 charts perfectly. In charts where it
makes errors, the models are never incorrect on all the questions. Both models
make large errors on a few data points, usually due to misreading a label.
Since these aren’t estimation errors, the range of errors is erratic, as
evident from Table 5. Both models struggle with charts where multiple figures
are present.

Table 5: Chart-wise breakdown for Labeled Category: GPT and Gemini Match Percentages and Error Ranges Chart Name | Gemini Match % | GPT Match % | Gemini Error Range | GPT Error Range
---|---|---|---|---
stacked bar chart | 100% | 80% | (0, 0) | (0, 5)
bar and table chart | 100% | 100% | (0, 0) | (0, 0)
piechart | 100% | 100% | (0, 0) | (0, 0)
simple table | 100% | 100% | (0, 0) | (0, 0)
icons on chart | 100% | 100% | (0, 0) | (0, 0)
stacked piechart | 100% | 100% | (0, 0) | (0, 0)
world map chart | 100% | 100% | (0, 0) | (0, 0)
bar chart diagonal labels | 100% | 75% | (0, 0) | (0, 2)
connected bar graphs | 86% | 100% | (0, 20) | (0, 0)
horizontal stacked bar chart | 80% | 100% | (0, 15) | (0, 0)
positive-negative bar charts | 80% | 80% | (0, 878) | (0, 178)
waterfall chart | 80% | 60% | (0, 194) | (0, 882)
eight multi-bar charts | 75% | 50% | (0, 1) | (0, 22)
two figures: stacked multi-bar | 67% | 67% | (0, 4) | (0, 4)
two figures - multi-bars | 20% | 40% | (0, 52) | (0, 46)
Overall | 86% | 84% |  |

##### GPT 4o failure points

In some cases, 4o misunderstands the data associated with a given label in
stacked charts. In ’two figures: stacked multi-bar’, when asked the Athleta
store count in 2012, it returned the count for Microsoft stores instead.
Similarly, in ’eight multi-bar charts’, it mistakenly returns the diabetes
rate for Japan instead of the obesity rate
In other cases, 4o misidentifies numbers on lower resolution charts. In
’stacked bar chart’, for instance, GPT 4o misreads ’3’ as ’8’ when asked to
identify the % of public sector publications by KPMG, a mistake that Gemini
did not make.

##### Gemini failure points

Similiar to 4o, Gemini errors are driven by misinterpreting the label but also
contain an element of misinterpreting the question, which GPT 4o doesn’t
demonstrate as frequently. In ’two figures - multi-bars’, for instance, when
asked what % of financing sources come from private investment (other than
EPR), it answers from the wrong bar graph. 4o on the other hand answers from
the correct table, but the wrong label within that table. While outside the
scope of these error rates, we also noticed that 4o was better at answering
some reasoning / calculation questions compared to Gemini

###  2.4 Unlabeled Charts: What is the magnitude of error across charts?

Across 82 questions in 15 unlabeled charts, both GPT 4o and Gemini Flash are
at par on the mean absolute percentage error at 55%.

Table 6: Chart-wise breakdown for Unlabeled Category: GPT and Gemini MAPE Chart Name | Gemini MAPE % | GPT MAPE %
---|---|---
stacked bar low res | 226 | 131
bubble chart | 147 | 177
two figures chart | 113 | 155
stacked bar and pie chart | 62 | 101
single line chart | 49 | 37
bar and line | 45 | 68
line on bar 2 | 39 | 37
simple line on bar | 35 | 8
tagged dots chart | 33 | 29
range line chart | 32 | 10
comparison bar charts | 29 | 24
seven line chart | 20 | 18
moon chart | 13 | 13
dots chart | 12 | 37
black line chart | 8 | 7
three line chart | 6 | 24
Overall | 52.79 | 54.64

##### Observations

Both Gemini and GPT 4o struggle with charts that have multiple unlabeled
figures, with both reaching >100% error rates (113% and 155% respectively).
This is driven by high magnitude errors across many/all of the questions for
that chart. Another common failure point is stacked charts, where the
estimation of a middle figure requires estimating all the labels in the chart
(error rates here reach upto 85% for a high resolution charts and a staggering
230% for a low resolution chart)
Both models also struggle with stacked bar charts, where estimating the number
may require isolating a sub-bar. In the ’stacked bar and pie chart’ for
instance, both models responded with the total value of the bar instead of the
value for a specific sub-bar label (50,000 vs. correct answer of 35,000)
Both models perform best on line charts with deviation rates as low as 6% on a
complex three line chart.
A detailed breakdown of the chart-wise questions and errors are given in the
appendix.

##  3 Conclusion

While both GPT-4o and Gemini Flash 1.5 exhibit many advanced capabilities in
reading charts, our evaluation reveals some consistent limitations in their
current performance. Despite being better than other alternatives, these
models still require human oversight to achieve acceptable accuracy levels.
For labeled charts, the models demonstrate an average error rate of 15%, which
may not be suitable for high-stakes business applications. The performance on
unlabeled charts is particularly inconsistent, with error rates exceeding 100%
for more complex visuals. When assessed using a comprehensive set of 30
charts, representing a typical business deck, the models are only able to read
7-8 charts with 100% match accuracy (all questions correct). Thus, for any use
case demanding high precision, these models are not yet ready to operate
without human intervention.

##  4 Appendix

###  4.1 Labeled Charts: Questions and Model Responses

Chart Type |  Question |  Correct Answer |  Gemini Flash Answer |  GPT 4o Answer
---|---|---|---|---
two figures: stacked multi-bar |  |  |  |
|  How many stores did Warby Parker open by 2013? |  13 |  13 |  13
|  In 2012, how many stores did Athleta have? |  35 |  35 |  31
|  In 2013, how many stores did Boston Proper have? |  4 |  8 |  4
|  In 2009, how many stores did Microsoft have? |  2 |  2 |  0
|  What is the % of capex to store related investments in 2012? |  71 |  71 |  71
|  What was the compound annual growth rate of the % of capex to store related investments from 2011 to 2013? |  6 |  4 |  6
|  Perfect Match % |  |  67 |  67
stacked bar chart |  |  |  |
|  What % of retail and distribution publications did PwC publish from 2013-15? |  6 |  6 |  6
|  How many industry publications did KPMG have from 2013-15? |  626 |  626 |  626
|  What % of retail and distribution publications did EY publish from 2013-15? |  4 |  4 |  4
|  What % of public sector publications did Deloitte publish from 2013-15? |  9 |  9 |  9
|  What % of public sector publications did KPMG publish from 2013-15? |  3 |  3 |  8
|  Perfect Match % |  |  100 |  80
connected bar graphs |  |  |  |
|  What is the projected total global fintech revenue (in $ billion) for 2030? |  1500 |  1500 |  1500
|  How much of the global fintech revenue in 2021 came from the banking sector? |  225 |  245 |  225
|  By how many times is the global fintech revenue expected to grow from 2021 to 2030? |  6 |  6 |  6
|  What is the projected fintech penetration of banking revenues (%) for 2030? |  13 |  13 |  13
|  What was the fintech penetration of banking valuations (%) in 2021? |  9 |  9 |  9
|  What is the projected revenue for the insurance sector in 2030 in billion dollars? |  200 |  200 |  200
|  What was the total global fintech revenue in 2021 in billion dollars? |  245 |  245 |  245
|  Perfect Match % |  |  86 |  100
eight multi-bar charts |  |  |  |
|  What is the prevalence of high blood pressure in France? |  28.5 |  28.5 |  28.5
|  How much more prevalent is diabetes in France compared to India? |  2.7 |  2.2 |  3.9
|  What is the prevalence of obesity in Brazil? |  22.3 |  22.3 |  22.3
|  What is the prevalence of obesity in Japan? |  4.4 |  4.4 |  26.7
|  Perfect Match % |  |  75 |  50
two figures - multi-bars |  |  |  |
|  What percentage of financing sources that supported the implementation of the solution came from private investment (other than EPR)? |  42 |  0 |  16
|  What is the percentage of governmental funding that supported the implementation of the solution? |  16 |  21 |  16
|  What percentage of financing for scaling the solution came from blended financing? |  21 |  0 |  21
|  How much did philanthropic funding contribute to the scaling of the solution in percentage terms? |  21 |  21 |  0
|  What is the combined percentage of multilateral funding for both the implementation and scaling of the solution? |  11 |  63 |  57
|  Perfect Match % |  |  20 |  40
bar and table chart |  |  |  |
|  What is the overall unbanked percentage? |  27 |  27 |  27
|  What is the cash usage in APAC? |  59 |  59 |  59
|  How many underbanked adults in Europe? |  357 |  357 |  357
|  How many more unbanked adults in APAC than in Europe? |  766 |  766 |  766
|  Perfect Match % |  |  100 |  100
piechart |  |  |  |
|  What percentage of TMT executives are ready to disclose Scope 1 emissions? |  69 |  69 |  69
|  What percentage of TMT executives are ready to disclose all other indirect emissions that occur in the company’s value chain? |  34 |  34 |  34
|  What percentage of executives overall are ready to disclose Scope 3 emissions? |  37 |  37 |  37
|  What is the difference in the % of TMT executives willing to disclose Scope 2 emissions and % of executives overall willing to disclose Scope 2 emissions? |  7 |  7 |  7
|  Perfect Match % |  |  100 |  100
horizontal stacked bar chart |  |  |  |
|  What % of the employee workforce is white in F&23? |  55 |  55 |  55
|  What % of the employee workforce is female in F&21? |  49 |  49 |  49
|  What % of the employee workforce is ’two or more races’ in F&22? |  3 |  3 |  3
|  What % of the employee workforce elected not to provide their race in FY21? |  4 |  4 |  4
|  What % of the employee workforce is Asian in F&23? |  22 |  7 |  22
|  Perfect Match % |  |  80 |  100
simple table |  |  |  |
|  What is the trades per month in market C? |  10 |  10 |  10
|  What are the net proceeds in market A? |  46 |  46 |  46
|  What is the price in Market B? |  48 |  48 |  48
|  What is the annual volume in market B? |  12000 |  12000 |  12000
|  What is the net proceed in market B? |  43 |  43 |  43
|  Perfect Match % |  |  100 |  100
icons on charts |  |  |  |
|  What was the NPS of Netflix? |  68 |  68 |  68
|  What was the NPS of Amazon? |  62 |  62 |  62
|  What was the NPS of Spotify? |  54 |  54 |  54
|  What was the NPS of Life? |  26 |  26 |  26
|  What was the NPS of Apple? |  68 |  68 |  68
|  Perfect Match % |  |  100 |  100
positive-negative bar charts |  |  |  |
|  What is the change in medical claims spent per participant (in $) for ’Hello Heart Users’ in the surgery/inpatient service category? |  -482 |  -482 |  -482
|  What is the change in medical claims spent per participant (in $) for ’Matched Control’ group in the diagnostics service category? |  580 |  -298 |  580
|  What is the change in medical claims spent per participant (in $) for ’Matched Control’ group in the physician visit service category? |  40 |  40 |  218
|  What is the reduction in total medical costs per participant per year (in $)? |  1865 |  1865 |  1865
|  What is the total change in medical claims spent per participant (in $) for ’Hello Heart Users’? |  -880 |  -880 |  -880
|  Perfect Match % |  |  80 |  80
waterfall chart |  |  |  |
|  What is the starting IH09? (in $ millions) |  1911 |  1911 |  1911
|  What is the deduction for investing activities (in $ millions)? |  97 |  -97 |  -97
|  What is the addition for cash capex (in $ millions)? |  441 |  441 |  -441
|  What is the pension contribution (in $ millions)? |  -187 |  -187 |  -187
|  What is the cash from operations amount (in $ millions)? |  285 |  285 |  285
|  Perfect Match % |  |  80 |  60
world map chart |  |  |  |
|  What is the % of women in the Americas? |  49 |  49 |  49
|  What is the number of people in Asia Pacific? |  56386 |  56386 |  56386
|  What is the number of people in EMA? |  142368 |  142368 |  142368
|  What is the total number of member firms? |  143 |  143 |  143
|  What is the overall % of women globally? |  48.5 |  48.5 |  48.5
|  What is the % of women in Asia-Pacific? |  54 |  54 |  54
|  Perfect Match % |  |  100 |  100
stacked piechart |  |  |  |
|  What is the % share of digital engagement in the personal care category? |  17 |  17 |  17
|  What is the % share of digital engagement in the computer software category? |  98 |  98 |  98
|  What is the % share of digital engagement in the food and drink category? |  3 |  3 |  3
|  What % of middle east and africa uses digital banking? |  17 |  17 |  17
|  What is the % share of digital engagement in the consumer banking category? |  58 |  58 |  58
|  Perfect Match % |  |  100 |  100
bar chart diagonal labels |  |  |  |
|  What is the accuracy % for resort in Graph Layer = 2 |  86.7 |  86.7 |  88.4
|  What is the accuracy % for viaduct in Graph Layer = 1 |  84 |  84 |  84
|  What is the accuracy % for forest in Graph Layer = 3 |  84.9 |  84.9 |  84.9
|  What is the accuracy % for forest in Graph Layer = 1 |  83.9 |  83.9 |  83.9
|  |  |  |
|  Perfect Match % |  |  100 |  75

###  4.2 Unlabeled Charts: Questions and Model Responses

Chart Type |  Question |  Correct Answer |  Gemini Answer |  OpenAI Answer
---|---|---|---|---
bar and line |  |  |  |
|  What was the highest revenue growth rate % seen post the financial crisis? |  11 |  10 |  8
|  What was the lowest revenue growth rate % seen before the financial crisis? |  5 |  10 |  8
|  What was the lowest R&D growth rate % seen before the financial crisis? |  10 |  5 |  5
|  What was the revenue growth rate in 2009? |  4 |  4 |  0
|  What was the revenue growth rate in 2019? |  11 |  10 |  6
|  What was the R&D growth rate in 2015? |  7.5 |  0 |  -2
|  Mean Absolute Error (MAE) |  |  3.25 |  4.92
|  Mean Absolute Percentage Error (MAPE) % |  |  44.70 |  68.23
bubble chart |  |  |  |
|  What is the change in disease burden between 2020 and 2040 for nutritional deficiencies? |  -20 |  -20 |  -20
|  What is the minimum change in disease burden between 2020 and 2040 for infectious diseases? |  -5 |  -40 |  -40
|  What is the highest increase in disease burden expected from a disease in the ’other diseases’ category? |  60 |  40 |  50
|  What is the change in disease burden between 2020 and 2040 for neglected tropical disease and malaria? |  -30 |  -30 |  -40
|  How many bubbles which represents an ’infectious disease’ have a change in disease burden that ranges from -20 to -40? |  3 |  3 |  4
|  |  |  |
|  Mean Absolute Error (MAE) |  |  11 |  15.20
|  Mean Absolute Percentage Error (MAPE) % |  |  146.67 |  176.67
line on bar 2 |  |  |  |
|  How many deals happened in 2018? |  30 |  10 |  20
|  How much funding (in millions) was deployed in 2021? |  1500 |  1500 |  1500
|  What was the deal count in 2019? |  40 |  40 |  20
|  How much funding (in millions) was deployed in 2017? |  100 |  10 |  0.5
|  What was the highest deal count between 2017 and 2022? |  110 |  110 |  120
|  What was the highest funding (in millions) between 2017 and 2020? |  1450 |  300 |  1000
|  |  |  |
|  Mean Absolute Error (MAE) |  |  210 |  98.25
|  Mean Absolute Percentage Error (MAPE) % |  |  39.33 |  37.16
tagged dots chart |  |  |  |
|  What % of survey respondents think Advanced IT will be a widely used skill in the future? |  35 |  26 |  30
|  What % of survey respondents think Basic IT will be a future skill needed? |  15 |  26 |  30
|  What percentage of respondents think ’Entrepreneurship’ will be an expected future skill needed? |  27 |  18 |  20
|  How many categories of skills are included in the limited but growing quadrant? |  3 |  4 |  3
|  How many skills are in the ’limited and stable’ quadrant of this chart? |  4 |  5 |  5
|  What percentage of respondents said that ’complex information processing’ will be an expected future skill needed? |  23 |  25 |  30
|  What percentage of respondents said that ’complex information processing’ is a most widely used skill today? |  22 |  15 |  20
|  |  |  |
|  Mean Absolute Error (MAE) |  |  5.71 |  5.29
|  Mean Absolute Percentage Error (MAPE) % |  |  33.03 |  29.25
moon chart |  |  |  |
|  What is the relative applicability of Machine Learning (AI) (as a %) in the negotiation stage of the procurement process? |  100 |  100 |  100
|  What is the relative applicability of Machine Learning (AI) (as a %) in the invoice checking stage of the procurement process? |  25 |  25 |  25
|  What is the relative applicability of Automation (as a %) in the payment processing stage of the procurement process? |  100 |  100 |  100
|  What is the relative applicability of Automation (as a %) in the RFX process stage of the procurement process? |  75 |  25 |  25
|  What is the relative applicability of cognitive agents (as a %) in the ’Buyer’ stage of the procurement process? |  25 |  25 |  25
|  |  |  |
|  Mean Absolute Error (MAE) |  |  10 |  10
|  Mean Absolute Percentage Error (MAPE) % |  |  13.33 |  13.33
black line chart |  |  |  |
|  What was the traction in Sep ’15 (in $ thousands)? |  60 |  55 |  60
|  What was the traction in Nov ’15 (in $ thousands)? |  75 |  80 |  80
|  What was the traction in Jul ’15 (in $ thousands)? |  48 |  40 |  60
|  What was the highest traction between May ’15 and May ’16 (in $ thousands)? |  140 |  130 |  137.3
|  What was the % increase in traction in the ’Last 30 days’? |  22 |  22 |  22
|  |  |  |
|  Mean Absolute Error (MAE) |  |  5.60 |  3.93
|  Mean Absolute Percentage Error (MAPE) % |  |  7.76 |  6.71
two figures chart |  |  |  |
|  What % of respondents say their frequency of use is 1-3 times a month because of airline miles? |  4 |  18 |  25
|  What % of respondents say their duration of most used credit card is 10 years or more because of low interest rates? |  23 |  10 |  20
|  What % of respondents say their frequency of use is 1-2 times a week because of unlimited cash back? |  14 |  13 |  10
|  What is the highest % of respondents who say their duration of most used credit card is 1 year but less than 2 years? |  12 |  28 |  35
|  What is the lowest % of respondents who say their frequency of use is 7-10 times a week? |  6 |  5 |  5
|  |  |  |
|  Mean Absolute Error (MAE) |  |  9 |  10.40
|  Mean Absolute Percentage Error (MAPE) % |  |  112.73 |  154.99
seven line chart |  |  |  |
|  What were the influenza rates in Feb 2016/17? |  28000 |  25000 |  30000
|  What were the influenza rates in Apr 2014/15? |  8000 |  6000 |  5000
|  What were the influenza rates in Jan 2015/16? |  10000 |  10000 |  10000
|  What was the average influenza rate in Mar between 2014-19? |  27000 |  15000 |  20000
|  |  |  |
|  Mean Absolute Error (MAE) |  |  4250 |  3000
|  Mean Absolute Percentage Error (MAPE) % |  |  20.04 |  17.64
single line chart |  |  |  |
|  What was the growth projection on 12/29/2003? |  80000 |  150000 |  100000
|  What was the growth projection on 5/17/2004? |  210000 |  400000 |  300000
|  What was the actual data on 4/19/2004? |  480000 |  400000 |  102000
|  What was the highest actual growth between 5/03 - Present? |  900000 |  900000 |  900000
|  |  |  |
|  Mean Absolute Error (MAE) |  |  85000 |  122000
|  Mean Absolute Percentage Error (MAPE) % |  |  48.66 |  36.65
stacked bar low res |  |  |  |
|  What were Origin’s ’load and bank guarantee - undrawn’ (in $ million) in 2011? |  800 |  1500 |  400
|  What were Origin’s ’US private placement’ (in $ million) in 2014? |  180 |  400.0 |  0.0
|  What were Origin’s ’load and bank guarantee - undrawn’ (in $ million) in 2010? |  90 |  400 |  200
|  What were Origin’s ’load and bank guarantee - drawn’ (in $ million) in 2012? |  1100 |  1500 |  200
|  What was Origin’s highest US private placement (in $ million) between 2010 and 2020? |  250 |  1600 |  1000
|  |  |  |
|  Mean Absolute Error (MAE) |  |  596 |  468
|  Mean Absolute Percentage Error (MAPE) % |  |  226.11 |  130.81
three lines chart |  |  |  |
|  What was the total return to shareholders by non resilient companies in 2015? |  120 |  120 |  150
|  What was the total return to shareholders by resilient companies in 2013? |  230 |  240 |  175
|  What was the total return to shareholders by resilient companies at the end of recovery phase? |  150 |  150 |  200
|  What was the total return to shareholders by S&P 500 companies at the end of growth phase? |  210 |  250 |  200
|  What was the total return to shareholders by S&P 500 companies at the end of downturn phase? |  75 |  70 |  100
|  What was the total return to shareholders by resilient companies in 2014? |  250 |  250 |  200
|  What was the total return to shareholders by non-resilient companies in 2010? |  80 |  90 |  100
|  |  |  |
|  Mean Absolute Error (MAE) |  |  9.29 |  34.29
|  Mean Absolute Percentage Error (MAPE) % |  |  6.08 |  23.62
stacked bar and pie chart |  |  |  |
|  What is the lead supply ’Today’ in Type 1 channel? |  35000 |  50000 |  50000
|  What is the lead supply in ’12 months’ in Type 1 channel? |  90000 |  100000 |  200000
|  What is the lead supply in ’12 months’ in Type 4 channel? |  25000 |  30 |  50000
|  What is the lead supply in ’Today’ in Type 4 channel? |  10000 |  20 |  50000
|  What % of loan distribution is with Lender 4? |  9 |  18 |  5
|  What % of loan distribution is with Lender 2? |  34 |  34 |  34
|  What % of loan distribution is with ’Other’ Lender? |  5 |  9 |  5
|  |  |  |
|  Mean Absolute Error (MAE) |  |  8566.14 |  27143.43
|  Mean Absolute Percentage Error (MAPE) % |  |  61.95 |  101.36
dots chart |  |  |  |
|  What’s the percentage of the population above empowerment line in Israel in 2022? |  73 |  70 |  75
|  What’s the GDP per capita 2022 in the US? (in $ thousands) |  75 |  65 |  70
|  What’s the GDP per capita 2022 in Ireland (in $ thousands)? |  105 |  90 |  80
|  What’s the percentage of the population above empowerment line in Ireland in 2022? |  73 |  70 |  75
|  What’s the highest percentage of the population above empowerment line for a country with empowerment line set at $12 PPP floor? |  40 |  50 |  100
|  |  |  |
|  Mean Absolute Error (MAE) |  |  8.20 |  18.80
|  Mean Absolute Percentage Error (MAPE) % |  |  12.17 |  37.19
comparison bar charts |  |  |  |
|  What was the instrument handling performance score when using Osso VR Training? |  4 |  3 |  5
|  What was the total performance score when using Standard training? |  7.5 |  8 |  6
|  What was the knowledge of instruments performance score when using Osso VR Training? |  3.5 |  2 |  5
|  What was the time and motion performance score when using standard training? |  2.5 |  1 |  2
|  What was the total performance score when using Osso VR training? |  17.5 |  19 |  20
|  |  |  |
|  Mean Absolute Error (MAE) |  |  1.20 |  1.40
|  Mean Absolute Percentage Error (MAPE) % |  |  28.62 |  24.43
simple line on bar |  |  |  |
|  What was the number of mocap systems (in thousands) in the market in 2024? |  75 |  87.5 |  80
|  What was the number of mocap systems (in thousands) in the market in 2022? |  15.0 |  20 |  20
|  What was the number of assets created by users (in million) per year in 2022? |  22.5 |  0.02 |  22.5
|  What was the number of assets created by users (in million) per year in 2023? |  44 |  40 |  45
|  What was the highest number of assets created by users (in million) between 2021 and 2024? |  80 |  67.5 |  80
|  |  |  |
|  Mean Absolute Error (MAE) |  |  11.30 |  2.20
|  Mean Absolute Percentage Error (MAPE) % |  |  34.93 |  8.45
range line chart |  |  |  |
|  What percentage of respondents prefer in-store experience during the purchase step of shopping journey for the furniture category? |  78 |  39 |  80
|  What percentage of respondents prefer in-store experience during the discovery step of shopping journey for the computer and electronics category? |  30 |  57 |  30
|  What percentage of respondents prefer in-store experience during the pickup step of shopping journey for the furniture category? |  35 |  38 |  30
|  What is the average in-store preference (percentage of respondents) during the trial step of the shopping journey? |  80 |  50 |  60
|  What is the lowest in-store preference % in the pickup step of the shopping journey? |  35 |  36 |  30
|  What is the highest in-store shopping preference % in the trial step of the shopping journey? |  85 |  80 |  80
|  |  |  |
|  Mean Absolute Error (MAE) |  |  17.50 |  6.17
|  Mean Absolute Percentage Error (MAPE) % |  |  32.47 |  10.34

## References

  * [1] B. Enhanced Chart Understanding in Vision and Language Task via Cross-modal Pre-training on Plot Table Pairs, arXiv, 2023. https://arxiv.org/pdf/2305.18641.
  * [2] D. Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysis, arXiv, 2023. https://arxiv.org/pdf/2308.11224.
  * [3] E. Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study, arXiv, 2023. https://arxiv.org/pdf/2305.13062.
  * [4] F. Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models - A Survey, arXiv, 2024. https://arxiv.org/pdf/2404.01869.
  * [5] Y. UniChart, "A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning", arXiv, 2023. https://arxiv.org/pdf/2305.14761.
  * [6] X. Zhang, Y. Xie, H. Zhang, et al. "A Survey on Evaluation of Large Language Models." arXiv preprint arXiv:2307.03109, 2023. Available: https://arxiv.org/pdf/2307.03109