o #1 spacing=nonfrench

# Educators’ Perspectives of Using (or Not Using) Online Exam Proctoring
††thanks: This is an extended version of a paper appearing at the 32nd USENIX
Security Symposium in August, 2023. Please reference:
David G. Balash, Rahel A. Fainchtein, Elena Korkes, Miles Grant, Micah Sherr,
and Adam J. Aviv. “Educators’ Perspectives of Using (or Not Using) Online Exam
Proctoring.” In the proceedings of the 32nd USENIX Security Symposium. USENIX
Sec’. Aug.2023.

David G. Balash, Elena Korkes, Miles Grant, and Adam J. Aviv
The George Washington University  Rahel A. Fainchtein and Micah Sherr
Georgetown University

###### Abstract

The onset of the COVID-19 pandemic changed the landscape of education and led
to increased usage of remote proctoring tools that are designed to monitor
students when they take assessments outside the classroom. While prior work
has explored students’ privacy and security concerns regarding online
proctoring tools, the perspective of educators is under explored. Notably,
educators are the decision makers in the classrooms and choose which remote
proctoring services and the level of observations they deem appropriate. To
explore how educators balance the security and privacy of their students with
the requirements of remote exams, we sent survey requests to over 3,400
instructors at a large private university that taught online classes during
the 2020/21 academic year. We had $n=125$ responses: 21% of the educators
surveyed used online exam proctoring services during the remote learning
period, and of those, 35% plan to continue using the tools even when there is
a full return to in-person learning. Educators who use exam proctoring
services are often comfortable with their monitoring capabilities. However,
educators are concerned about students sharing certain types of information
with exam proctoring companies, particularly when proctoring services collect
identifiable information to validate students’ identities. Our results suggest
that many educators developed alternative assessments that did not require
online proctoring and that those who did use online proctoring services often
considered the tradeoffs between the potential risks to student privacy and
the utility or necessity of exam proctoring services.

##  1 Introduction

The initial surge of the COVID-19 pandemic upended education, leading many
schools to quickly switch to remote teaching in the Spring of 2020 [4], and
many universities and colleges maintained remote learning into the 2020/21
academic year. This massive migration to online learning environments led to a
corresponding increase in the use of remote educational technologies.

One such remote learning technology that saw a dramatic increase in use during
remote instruction is online exam proctoring tools. Based on the analysis of
the Chrome browser extension reviews, Balash et al. found explosive growth
(720%) of online proctoring beginning at the start of the COVID-19 pandemic
[2]. This is in line with a poll by Grajek that found that 77% of colleges and
universities made use of or were planning to use online proctoring [12].

By design, remote proctoring systems are invasive. Given their capabilities to
monitor and limit functionality where installed, students and privacy
advocates have raised concerns about their security and privacy properties. As
highlighted by the media coverage of remote proctoring tools, these concerns
were not unfounded: Since the tools’ widespread adoption at the beginning of
the pandemic, reports uncovered major security and privacy incidents involving
Proctorio and ProctorU, two widely used invigilation tools. These included a
major data breach of ProctorU in which 444,000 users’ personally identifying
information was leaked online and a security vulnerability within Proctorio
that allowed hackers to remotely activate the software on computers in which
it was installed [1, 29, 27]. More recently, Burgess et al. [3] disclose
several security and privacy issues, including concerns about how remote
proctoring systems use facial recognition.

In a survey of students who experienced remote proctoring, Balash et al. found
that many students had both privacy and security concerns with the tools [2].
In particular, student participants often felt they had no choice but to use
the tools or that they trusted these proctoring services because of their
academic institutions’ support for them. Despite some students’ trust in these
tools’ security, Coheny [7] show evidence indicating that these tools may not
be as trustworthy as students suspect. Specifically, they find that many
collaborative tools used in remote classrooms collect information about
students that often does not align with educational expectations. However, the
tools analyzed in their study focus on collaboration tools that were not
designed for academic settings.

As such educators’ perspectives of online exam proctoring services remains
unexplored. Educators‘ perspectives are of particular importance given their
roles in both the choice to use (or not use) an online proctoring tool and its
associated monitoring. In this paper we seek to answer the following research
questions about how educators consider privacy and security in the context of
online proctoring services:

  1. RQ1

What are educators’ perceptions of online proctoring services?

  2. RQ2

Do educators consider student privacy and security concerns when deciding to
use (or not use) an online proctored exam and while setting up the exam
proctoring parameters?

  3. RQ3

Which proctoring methods do educators select to proctor their online exams?

To answer these research questions, we executed a campus-wide recruitment of
instructors at the George Washington University who taught courses during the
remote learning period of the 2020/21 academic year. This involved inviting
3,460 educators to participate in an IRB-approved survey, of which $n=125$
participants responded with their justifications for using or not using online
proctoring. The survey captured responses from the university’s 12
organizational units, senior and junior faculty, as well as graduate
educators. Despite our small sample (approximately $\nicefrac{{1}}{{5}}$ of
respondents opted to use online proctoring), our results offer important and
timely insights into how educators at a large-private university understood
online proctoring tools and their motivations for using (or not using) these
tools during a challenging period.

We found that a small but substantial number (21%) of educators used exam
proctoring tools during the 2020/21 academic year. The most common reasons for
using remote proctoring were to stop or deter cheating, to comply with
COVID-19 safety protocols, to maintain exam integrity, and to be fair to
students. In contrast, 79% of respondents did not use online exam proctoring
tools during this same period. Many chose not to use remote proctoring tools
due to their potential harms to students, negative impacts on trust between
students and educators, student privacy concerns, ineffectiveness against
cheating, and the availability of alternative modes of assessment, such as
open-book exams and projects.

Both educators who used online proctoring and those who did not reported
privacy concerns with using exam proctoring services. These concerns centered
on webcam and audio recordings taken by a third party, intrusive monitoring
measures, information sharing requirements, particularly those of personally
identifying information for verification of student identities, and the
invasion of student privacy as students take these exams in their homes.
Educators also expressed concerns about the security implications of students
having to install exam proctoring software on their computers. Many
highlighted the software’s monitoring capabilities and its ability to disable
system functionality.

While many educators chose to modify their assessments rather than use online
exam proctoring, some educators were required to use online exam proctoring
tools. Specifically, these educators reported departmental mandates for the
use of exam proctoring services in their courses, or requirements to
administer the standardized tests in their field, such as nursing, that were
only offered by testing companies that use online exam proctoring
technologies. This led to educators being forced to use these proctoring tools
despite having reservations about their use and concerns about their impact on
students.

##  2 Related Work

Online proctoring tools have been the subject of heavy scrutiny from both the
media [28, 17, 22, 15] and education researchers [31, 16, 10, 20, 26, 24].
Below, we frist review the literature on the whether proctoring is needed to
ensure academic integrity and are efficacious in doing so, and how they impact
students have been at the center of debates on their role within remote
learning. Following, we will discuss more recent work studying the security
and privacy impact of this technology.

##### Effectiveness in Academic Integrity

When it comes to the role of online proctroing in online learning, Harton et
al. find that despite vast improvements in remote learning tools, university
instructors and students show a strong bias towards the beliefs that online
courses are more conducive to academic dishonesty and that cheating occurs
more often in online settings [14]. However, studies comparing the rates of
academic dishonesty in face to face courses to its prevalence in online
courses have found mixed results: Watson and Sottile find that while students
more readily admit to academic dishonesty in face to face classes, they are
more likely to cheat during online exams [30]. In contrast, Grijalva et al.
[13] find no significant difference in rates of academic dishonesty in both
course formats.

Gudiño Paredes et al., who evaluated remote proctored exams’ usage in graduate
learning via a questionnaire-based study, find that the tools enforce academic
integrity, but that students’ honesty is neither driven by their moral
compasses nor by their desire to learn[21]. Instead, as Gudiño Paredes et al.
explain, students lack opportunities to cheat due to constraints implemented
within the tools and feel obliged to behave with integrity lest they be caught
and punished by the software. Moreover, they find that these tools appear to
negatively impact students’ learning or motivation to learn and raise concerns
about student privacy. They therefore recommend educators carefully consider
their motivation to use remote proctoring before choosing to use these tools.

##### Student Performance Under Proctoring

Several studies [11, 8, 25, 9] have found that student performance was
significantly better on unproctored remote exams than on remotely proctored
ones. Seife and Stockton [25] further find that scores on remotely proctored
exams are more closely correlated with predictive attributes of student
performance, such as their ratings of human capital, which measures their
general ability level. This, they argue, shows evidence that academic
misconduct is likely quite pervasive, and that this pays off handsomely for
dishonest students. Moreover, Wuthisatian[32] finds that students generally
performed better on in-person exams than they did when the same exam was taken
with remote online proctoring.

In contrast however, Hylton et al. do not find a significant difference in
student performance between remote exams that use video monitoring and those
that do not. Despite this, they note that students in non-video monitored
exams took longer to complete their exams on average[18]. Rios and Liu
similarly find that student performance on low stakes exams is not impacted by
the use of online proctoring. However, unlike Hylton et al., they do not
observe any differences in the amount of time students take to complete their
exams[23]. In a study of the differences between testing centers and remote
proctoring by Cherry et al. [5], average scores achieved across the two
proctoring modes were similar.

##### Privacy, Security and Ethics of Remote Proctoring

Online remote proctoring and other online learning technology have received
considerable attention due to the pandemic. Coghlan et al. [6], in their
opinion piece, highlight proctoring tools’ reliance on artificial intelligence
and the ethical challenges this can raise. They explore an ethical framework
for determining when and how to use these tools. In their opinion article,
Swauger argues the underlying algorithms for remote monitoring and
invigilation contain implicit negative biases and that these tools unfairly
penalize students who do not meet their biased baseline[28].

Despite these concerns, research on these tools’ security and how users
perceive their security has been sparse. Balash et al. and Kharbat and Abu
Daabes independently study student perceptions of the tools, and find a high
prevalence of privacy concerns [19, 2]. Balash et al., who also analyze the
security and privacy of several tools, specifically focus on how students’
security and privacy concerns compare with the security vulnerabilities they
encounter[2]. Neither Balash et al. nor Kharbat and Abu Daabes consider the
perceptions of educators, the focus of this paper.

Recent work by Burgess et al. [3], performed a technical S&P analysis of four
proctoring suites used in high stakes law licensing exams, such as the Bar
Exam and entrance exams. They identified numerous privacy and security risk,
including around facial recognition. In this paper, we investigate the
educator’s perspective on eight general purpose exam proctoring software
suites, which are non-overlapping with the suites studied by Burgess et al..
Such a technical investigation, though, would likely be fruitful of the common
university remote exam proctoring products.

Similar to this paper is the study by Cohney et al. that also considers
instructor and faculty perceptions of remote learning tools [7], but not
specifically remote proctoring tools. The focus of Cohney et al. is on the use
of tools that were not originally designed for educational use, but were
adopted hastily amidst the pandemic to accommodate the need for fully remote
learning. This includes using Zoom, Google Drive, and other collaboration
platforms whose privacy standards and data collection practices do not match
the expectations of the classroom. In contrast, in this paper, we focus on
instructors perceptions, and more specifically, on educators’ perceptions of
remote proctoring tools and their security and privacy attributes, both why
they choose to use them and why not. Cohney et al., in contrast, identifies
such tools, but was not the primary focus of the study.

##  3 Survey Methodology

We conducted an online survey to evaluate university educator perceptions of
online exam proctoring tools. Here we describe the survey’s procedures,
recruitment, limitations, and ethics. Survey results are presented in Section
4.

##### Study Procedure

Below we outline the survey. The full text can be found in Appendix A.

  1. 1.

Informed Consent: The university educators were asked to consent to the study.
The consent included that participants would answer questions about their
experience with online exam proctoring services.

  2. 2.

Eligibility Screening: To be eligible to complete the survey, participants
were required to assert that they were either full-time faculty or part-time
adjunct faculty at the university.

  3. 3.

Background: The educators were then asked to optionally provide their
associated organizational unit or school at the university, as well as the
subject area(s) they taught during the 2020/21 academic year

  4. 4.

Awareness of Technology: The university educators were asked about their
awareness of the online exam proctoring tools available at the university
during the 2020/21 academic year and their understanding of how online exam
proctoring tools work. Next participants were asked which specific online exam
proctoring tools, if any, they used in administering assessments during the
2020/21 academic year.

  5. 5.

Use and Perceptions of Online Exam Proctoring Tools: Educators were asked
which proctoring services they most recently used, what factors they
considered, and the type and number of assessments administered with online
proctoring. Next the educators were asked about both the benefits and
drawbacks of online exam proctoring and under what conditions they were likely
to use online exam proctoring in the future.

  6. 6.

Proctoring Effectiveness: We then asked the educators about the effectiveness
of online exam proctoring tools at preventing and catching cheating on
assessments.

  7. 7.

Review of Proctoring Tools Used: Educators were asked questions to assess the
specific exam proctoring tools they reported to have recently used during the
2020/21 academic year. This included questions about the educators’ views of
the privacy and security of the online exam proctoring software, its
effectiveness, and the potential tradeoff between student privacy concerns and
the integrity of the examination being administered.

  8. 8.

Online Exam Proctoring Methods: In this part of the survey we investigate the
methods used by online exam proctoring services to monitor student test
takers. Educators were asked which exam monitoring methods they enabled in
their proctored exams, the effectiveness of these methods, their comfort using
the methods, and if they would change methods for future exams.

  9. 9.

Privacy Concerns: Finally, the survey concluded by asking educators about
their concerns for their students’ privacy when students are required to share
information with exam proctoring companies.

##### Recruitment

We worked with the George Washington University’s administration to approve
and coordinate the survey, in addition to receiving IRB approval (NCR202908).
In turn the university provided our research team with an email list of all
instructors who had taught a course during the 2020/21 academic year, totally
3,460 individuals. Recruitment occurred over a fifteen day period starting
December 1st 2021. We sent out 3,460 emails and had 152 educators respond to
the study, a response rate of 4.4%. Of the 152 educators who responded, 125
completed the study. Recruitment emails was sent by and the survey was hosted
on the university’s Qualtrics account. Participants who completed the survey
were given the opportunity to enter a drawing for a $50 USD Amazon gift card
with a 1 in 20 chance of winning. On average, it took 16.3 minutes (SD=37.4)
to complete the study.

Note that many of the instructors who were contacted may either no longer be
at the university or did not teach classes involving exam or quiz assessments
(e.g., instead using grading based solely on term-papers) that would render
them eligible to complete the survey. These instructors likely self-selected
out of the survey, and thus the true number of eligible participants and the
true response rate to the survey is difficult to determine. However, we
believe we captured a reasonably representative cross-section of the
university’s instructors during this period, both in terms of educators who
chose to use online proctoring and those who chose not to use it. But it is
also important to acknowledge that there are likely some perspectives that may
be over- or underrepresented due to self-selection both to take and not take
the survey.

##### Analysis Methods

When presenting quantitative results the analysis is provided in context. For
qualitative responses, we conducted open coding to analyze 14 free-response
questions. A primary coder from the research team crafted a codebook and
identified descriptive themes by coding all responses to each question. For
the 10 open-ended questions answered by participants who did use exam
proctoring tools, a secondary coder (also a member of the research team) coded
all responses. For the 4 open-ended questions responded to by educators who
did not use proctoring tools, a secondary coder coded a 20% sub-sample as a
consistency check. In each case, the secondary coder provided feedback on the
codebook, and inter-rater reliability was calculated on each round until
Cohen’s $\kappa\geq 0.7$. Overall, the mean Cohen’s $\kappa=0.8$, indicating
substantial agreement between coders. In the results presented below, we use
the primary-coders application of the final codebook for any counts or themes
presented.

##### Ethical Considerations

The study protocol was approved by our Institutional Review Board (IRB) with
approval number NCR202908, and all collected data is associated with random
identifiers. For participants who wanted to enter the drawing for a chance to
win the Amazon gift card we created an entry form that was separate and not
linked to the survey.We also considered that some educators may not want to
share how they managed student academic integrity for online classes or their
specific academic department or subject area and so we made those questions
optional.

##### Limitations

Our study is limited in its recruitment, particularly instructors at a single
academic institution in the U.S. While our study offers a unique perspective
at our institution, which is a large private university, we cannot claim full
generalizability of the results within or beyond the institution as it is
difficult to know the true number of eligible participants who considered or
used online proctoring during the 2020/21 academic year. Despite this
limitation, we believe that these results offer new insights and are likely
representative of common attitudes and themes among educators about online
proctoring and their choices to use or not use these products. However, we
cannot conclude that the these themes occur at the same proportions beyond our
sample. We attempt to note this limitation throughout when discussing
proportional results. This is particularly true for those instructors who
indicate that they do use online proctoring, and we cannot be confident that
quantitative results, e.g., Likert responses, will be consistent in a larger
sample. Although qualitative themes likely express dominant views in this
subgroup, we may not capture all minority themes. Throughout the following
section we acknowledge these limitations.

There is also limitations with the size of our recruitment. We got approval to
send recruitment emails to all instructors during the online-instruction
period at our institution, which included 3,460 individuals. Even within this
pool, finding college level educators that use exam proctoring turns out to be
a difficult-to-reach population. As with any online survey without direct
recruitment, response rates can be small, and then within those responses, we
are further seeking a set of educators who used online proctoring. We
acknowledge that the result is a sample of educators who actually used
proctoring is smaller than desirable, but when targeting hard-to-reach
populations (namely, college educators that use exam proctoring), exploratory
studies like this one, even with smaller samples, provide important and
relevant themes.

Importantly, the goal of this study is not only on the educators who did use
proctoring, but also those that choose not to and their security and privacy
reasons for that choice. We were able to recruit 99 participants who decided
against using online exam proctoring tools to provide important insight into
their decision making processs.

Finally, we are limited by the fact that this study relies on self-reported
behavior. We cannot verify that the participants actually used remote
proctoring tools to proctor an online exam or which monitoring methods they
enabled. Finally, responses can suffer from social desirability and response
bias, leading participants to over describe their awareness of online exam
proctoring as they may believe that this is the expectation of the
researchers. Such biases may be most present when participants indicate
concerns.

##  4 Results

All of the educators in our study taught a course during the 2020/21 academic
year at a the George Washington University. Educators from twelve of the
university’s organizational units or schools were represented (see Table 1),
with the largest percentages from the College of Arts & Sciences ($n={49}$;
39%), School of Public Health ($n={16}$; 13%), School of Medicine ($n={12}$;
10%), and the School of Business ($n={11}$; 9%) (Q2).

During the 2020/21 academic year the educators surveyed taught a wide range of
subjects. The most common of which were science, technology, engineering, and
mathematics ($n={40};~{}32\%$), health ($n={30};~{}24\%$), business
($n={14};~{}11\%$), and government ($n={12};~{}10\%$) (Q3). For the full
results see Table 2. Twenty-one percent ($n={26}$) of educators who responded
to our survey used online exam proctoring tools to assist in administering
assessments during the 2020/21 academic year (Q6).

At the time of the study, eight online exam proctoring tools were available at
the university. Educators reported being most aware of exam software by
Respondus ($n={56};~{}45\%$), ProctorU ($n={16};~{}13\%$), and Examsoft
($n={9};~{}7\%$) (Q4). Of the educators who reported using exam proctoring
software, the largest number reported using Respondus ($n={15}$ of
${26};~{}58\%$), followed by RPNow ($n={3}$ of ${26};~{}12\%$), Examsoft
($n={2}$ of ${26};~{}8\%$), and Proctorio ($n={2}$ of ${26};~{}8\%$), for
their most recent proctored online exam (Q7). Most ($n={23}$ of
${26};~{}88\%$) educators who used online exam proctoring tools used them for
administering course exams (e.g. test, midterm exam, final exam) (Q9). Among
educators who used online proctoring sixty-five percent reported ($n={17}$ of
${26}$) having administered five or more online proctored assessments (Q10).

Table 1: The number of educators in each of the organizational units (Q2), the number of those educators who used online exam proctoring tools (Q6), and the number of those educators who considered using the tools (Q6N1). Organizational Unit | Educators | Used | Considered
---|---|---|---
Arts & Sciences | 49 | 9 | 14
Public Health | 16 | 0 | 3
Medicine | 12 | 7 | 1
Business | 11 | 3 | 2
International Affairs | 8 | 0 | 1
Engineering & Applied Sci. | 8 | 1 | 2
Nursing | 6 | 5 | 1
Education | 6 | 0 | 0
Professional Studies | 3 | 0 | 0
Other | 3 | 1 | 0
Political Management | 1 | 0 | 0
Public Affairs | 1 | 0 | 1
Arts & Design | 1 | 0 | 0
Total | 125 | 26 | 25
Table 2: The number of educators in each subject (Q3), the number of those educators who used online exam proctoring tools (Q6), and the number of those educators who considered using the tools (Q6N1). Subject | Educators | Used | Considered
---|---|---|---
S.T.E.M. | 40 | 5 | 12
Medicine & Health | 30 | 8 | 5
Business | 14 | 6 | 2
Government | 12 | 2 | 2
Did not disclose | 7 | 3 | 0
Arts | 5 | 0 | 1
History | 5 | 0 | 0
Languages | 4 | 0 | 1
Communications | 4 | 1 | 1
Gender Studies | 1 | 0 | 1
Law | 1 | 1 | 0
Naval Science | 1 | 0 | 0
Teaching | 1 | 0 | 0
Total | 125 | 26 | 25

###  4.1 RQ1: Educators’ Perceptions

##### Educator Understanding of Exam Proctoring Tools

We asked educators to describe in their own words how online proctoring tools
work (Q5). Many ($n={65}$) educators described the ways the proctoring tools
monitor a student’s activity and behavior during an exam. Educator P2 (College
of Arts & Sciences) responded, “Monitor student’s actions and movements (and
room content) to make sure they are not cheating on an exam.” Educators
($n={30}$) also detailed how the proctoring tools restrict a student’s
activities and access to unauthorized resources. Educator P41 (College of Arts
& Sciences) explained, “The software takes control of a students computer so
that they can’t leave the exam, can’t access the internet, and can’t access
other programs on the computer.” Additionally, educators ($n={19}$) described
how proctoring tools record and flag anomalies during the exam taking session.
For example, educator P114 (School of Public Health) added, “Software can also
record the user (visual and audio) while the user is taking the test and can
flag any suspicious activity (user getting up from the computer, looking down
at table, etc.) for the instructor to later review to determine whether
cheating occurred.” However, only a few ($n={3}$) educators described the
ability of the proctoring service to verify a student’s identity. P44 (College
of Arts & Sciences; Respondus) noted, “There are various ways that they can
monitor student identity while taking the exam (web cam).”

Some educators ($n={32}$) reported that they did not know how they work.
Educator P16 shared, “I don’t know anything about how they work.” Others
($n={10}$) had simply not used them or were not aware of there availability at
the university.

##### Reasons for Not Using Online Proctoring Services

Of the 79% ($n={99}$) of educators responding to the survey that did not use
online exam proctoring tools, 25% ($n={25}$ of ${99}$) considered doing so at
some point (Q6N1). The College of Arts & Sciences had highest percentage (30%;
$n={12}$ of ${40}$) of educators who considered using exam proctoring, and
likewise, S.T.E.M (29%; $n={14}$ of ${49}$) was the highest percentage subject
matter (see Tables 1 and 2). (Q6N2). For instance, P77 (College of Arts &
Sciences) who selected _Somewhat unlikely_ to use online exam proctoring
(Q6N5) explained,

> Cheating during exams is a serious (and somewhat common) issue. Being
> assured that cheating was kept to a minimum would provide confidence that
> grades were well-earned.

For many of the 73% ($n={72}$ of ${99}$) of educators who did not consider
using online proctoring tools, their decision was likely informed by their
negative perceptions of these tools Q6N3). Many ($n={24}$) reported that they
considered online exam proctoring tools to be harmful to students. Educator
P10 College of Professional Studies, _Extremely unlikely_) stated, “Proctoring
tools monitor the students in ways that increase their anxiety and obliterate
their ability to learn.” Likewise, educator P77 (College of Arts & Sciences,
_Somewhat unlikely_) shared,

> Privacy and home issues. Not all students were in a position to be engaged
> in being a student while remote learning. If a student was a primary care
> giver for a child or elderly person, how could I penalize them from looking
> away from their screen during an exam?

P10 (College of Professional Studies, _Extremely unlikely_) even described it
as “prison technology,” and others ($n={11}$) concluded that it impacts the
trust between student and educator. For instance, educator P81 (College of
Arts & Sciences, _Somewhat unlikely_) noted, “It feels invasive and I feel it
erodes trust between student and professor.” Furthermore, educators ($n={10}$)
had concerns about the negative impact on student privacy, such as when
educator P76 (College of Arts & Sciences, _Extremely unlikely_) shared, “I
found the on-line proctoring system to be a serious invasion of privacy of the
student.”

Some ($n={9}$) educators determined that online proctoring tools lacked the
ability to actually stop cheating, like when P53 (College of Arts & Sciences,
_Extremely unlikely_) said, “…just locking down a computer is meaningless when
students can easily access a second computer (or their phone).” There were
also a number ($n={16}$) who were not aware of the availability of the tools
at the university. Educator P16 (College of Arts & Sciences, _Neither likely
nor unlikely_) noted, “I have no knowledge of them or their availability.”

Many educators refactored their assessment formats to avoid the necessity of
online exams (Q6N4). A common tactic was to provide time limits enforced
through existing learning management software, such as Blackboard. For
example, educator P96 (College of Arts & Sciences, _Extremely unlikely_)
stated, “I gave exams on Blackboard. They were timed, so that students would
have limited time to look up answers.” Some educators ($n={14}$) reported
changing their exams to open book and open note exams, such as educator P90
(College of Arts & Sciences, _Somewhat unlikely_) who explained, “I ended up
making everything open book so that I did not have to police anything.” Others
switched to take home assessments, like P14 (School of International Affairs,
_Somewhat unlikely_) who added, “During Covid I made the quizzes take home and
open book.” Replacing the exams with other forms of assessment, such as
projects and written papers, was another common theme ($n={34}$). For example,
educator P41 (College of Arts & Sciences, _Somewhat likely_) said, “I decided
to replace my exams (midterms and finals) with two-week take-home projects,
with many scaffolded layers.” Another example is educator P109 (School of
Business, _Somewhat likely_) who shared, “I ditched the quizzes and tests,
opting instead for graded homework and written papers.” Still others ($n={4}$)
reduced the percentage of the overall course grade which would come from
exams, like P93 (School of Engineering & Applied Science, _Neither likely nor
unlikely_) who illustrated, “The only way I actively managed it was 1. to put
tremendous credit on the term project and 2. reduced credit for exams.”

Another common theme ($n={9}$) among educators who chose not to use online
exam proctoring tools was a belief that students would not cheat when asked to
adhere to the university code of academic integrity. For instance, P76
(College of Arts & Sciences, _Extremely unlikely_) stated,

> I told students that I expected them to be adults, and to follow the
> university expectations of integrity and honesty. This was after I told them
> my opinions of the proctoring system to be an invasion of their privacy.
> They appreciated my opinion and cooperated with taking exams with honesty.

Educator P100 Navy ROTC, _Somewhat unlikely_) trusted students to follow the
university honor code and noted, “I reminded my students of the honor code and
trusted them to follow it.”

Finally, it appears that most educators responding to the survey ($n={53}$ of
${99}$; 54%) who do not currently use online exam proctoring tools in their
classes reported they would be unlikely to use them if they were teaching
remotely under similar circumstances to those of the 2020/21 academic year,
while only 24% ($n={24}$ of ${99}$) said they were likely (Q6N5). This
suggests that those who declined to use remote proctoring are unlikely to
change their opinion of the technology. When describing why they choose not to
use online exam proctoring tools (Q6N3), the participants who reported being
_Extremely unlikely_ or _Somewhat unlikely_ to use the tools in the future
(Q6N5), more often described themes such as proctoring tools as potentially
harmful to students (20 of 53 vs. 3 of 24), privacy concerns (10 of 53 vs. 0
of 24), tools do not stop cheating (9 of 53 vs 0 of 24), and trust students
not to cheat (9 of 53 vs. 2 of 24).

##### Reasons for Using Online Proctoring Services

Twenty-six educators responding to the survey reported using online proctoring
services. We asked them what factors they considered when deciding to use
these tools in an open response question (Q8). These factors may include
majority opinions; however, they may not capture minority opinions due to the
small number ($n=26$) of educators who used these services.

The most cited reason for using online proctoring tools is the convenience
they offered ($n={12}$). For many ($n={7}$) this convenience was attributed to
their familiarity with the proctoring tools either because they (or their
colleague) have previously administered an online proctored exam using the
tool ($n={6}$). For instance, P69 (College of Arts & Sciences; Respondus)
noted, “Familiarity based on discussions with colleagues (who all used
respondus or proctor exams themselves)” and P25 (School of Nursing; Proctorio)
added, “Already using this product - the proctoring version for the online
environment is called Examplify (w/ ExamSoft).” Others ($n={4}$) were
influenced to use online proctoring due to their apparent popularity and
recommendations from others, like P67 who simply said, “It was popular.” A few
($n={2}$) educators noted that these tools had been recommended to them by
their institution, such as P26 (College of Arts & Sciences; Respondus) who
recalled, “It was recommended by the school.”

Many educators ($n={8}$) noted that their main motivation to use remote
proctoring was out of a form of necessity. Specifically, most ($n={6}$)
indicated that they were required to use online proctoring by their department
in an attempt to make assessment more uniform. For example, P33 (School of
Medicine & Health Sciences; RPNow) said,

> RPNow is the one used by my department. I don’t believe that I have a choice
> of which online proctoring service to use. It is already set up in my
> courses for me.

Or because they were administering a standardized test ($n={2}$), like P25
(School of Nursing; Proctorio) who stated, “Nursing students also take
standardized exams via ATI - their proctoring service in an online environment
is called Proctorio.” Some educators ($n={2}$) felt compelled to use exam
proctoring due to the circumstances of the COVID-19 pandemic and remote
learning, e.g., P36 (School of Nursing; Examsoft) who stated, “Absence due to
Covid exposure.”

We further queried educators who indicated that they used exam proctoring in
open-responses regarding the benefits of using these tools (Q11). The most
frequently ($n={16}$) mentioned benefit was to to enforce exam rules or exam
integrity as the primary benefit using online proctoring. Specifically,
educators indicate that the tools help prevent cheating ($n={10}$), protect
the integrity of remote exams ($n={2}$), and ensure exam fairness, by limiting
the benefit, or competitive edge students can gain from cheating ($n={2}$).
For example, P33 (School of Medicine & Health Sciences; RPNow) describes how
online exam proctoring tools deter cheating as a primary benefit:

> Even if not activated, students go through the [remote proctoring] system to
> take their exams, so they are under the impression that they are always
> being monitored. Ensures integrity of the exam without having to re-write
> questions to be open book.

Educators also felt that a benefit of online proctoring was to enforce exam
rules by verifying students’ identities ($n={1}$), and holding students
accountable for any misconduct they may commit ($n={2}$). As P123 (School of
Medicine & Health Sciences; RPNow) describes, “[The tools] provide a permanent
record of the student’s behavior during an exam.” Other educators ($n={2}$)
highlighted how remote proctoring tools allowed them to enforce other exam
rules, like time limits ($n={1}$) and prohibit access to prohibited resources
($n={1}$), e.g. P26 (College of Arts & Sciences; Respondus) added, “Ensuring
that students don’t use web resources to complete the test.”

Many ($n={13}$) educators noted as a benefit how online proctoring tools
offered additional flexibility. As ($n={11}$) instructors explained, these
tools made it easy for them to set up and grade their exams. While ($n={5}$)
respondents highlighted the tools’ general ease of use, e.g. P21 (School of
Nursing; ProctorU) noted, “Ease of use can be done online.” Additionally,
instructors found it allowed them to give proctored exams while complying with
COVID safety precautions, such as when P118 (School of Nursing; Proctorio)
stated, “Convenience and safety during COVID.”

Finally, a handful of educators noted that online proctoring made it easier to
manage their exams when compared to proctoring exams in person ($n={2}$), that
the proctored were easier to grade ($n={1}$), and that they were convenient to
use ($n={2}$). One educator noted the flexibility of using multimedia content
in their exams when online, and others $(n={2})$ noted that it also provides
flexibility to students in selecting the time and environment for their exam.
For instance, P35 (School of Medicine & Health Sciences; Respondus) shared,
“It gave flexibility to the students to take the exam when convenient instead
of at a set time.”

##### Drawbacks when Using Online Proctoring Tools

We also asked educators who indicated they used online proctoring tools
($n=26$) the drawbacks of online proctoring (Q12). These drawbacks may include
majority opinions; however, they may not capture minority opinions due to the
smaller number ($n=26$) of educators who used these services.

Most educators ($n={20}$) identified at least one technology or usability
issue they encountered while doing so. Chief amongst the drawbacks were
technology glitches ($n={7}$), and system limitations ($n={17}$) that hindered
students’ or educators’ ability to use the tools for their intended functions.
They noted that these limitations impacted their ability to monitor students
during exam time, to control students’ test environment and ensure academic
integrity, and to conclusively identify cases where students had cheated. A
cited cause ($n={3}$) for these issues were limitations on students’ computers
to run the proctoring software or students’ internet access. When it came to
connecting to the proctoring tools, instructors noted that some students
either had unstable internet connections ($n={1}$), or had limited access to
their exams due to being located in a different country ($n={1}$). In other
cases, students’ computers seemed to be the point of failure. In particular,
instructors noted that some students’ sometimes used older computers
($n={1}$). This meant that their machines would occasionally freeze when
running the proctoring tools ($n={1}$), that they would not have webcams or
microphones through which their exam session could be recorded, or that these
input devices would fail to record ($n={1}$) while students took their exams.
As P44 (College of Arts & Sciences; Respondus) explains, the software’s lack
of dependability posed a significant “obstacle for students.”

Educators also noted drawbacks with respect to the privacy of their students.
Several ($n={6}$) cited concerns for their students’ privacy. In particular,
they noted wariness about third parties potentially collecting vendor data
about their students ($n={1}$), and that they found monitoring via video or
audio recording to be privacy invasive ($n={3}$). As P33 (School of Medicine &
Health Sciences; RPNow) explained they “[Felt] uncomfortable seeing students’
living situation and watching them while taking the exam.” (We elaborate more
on the privacy concerns in subsection 4.2.)

Additionally, two educators also described drawbacks with respect to the
interpersonal relationship with students that subjecting them to online
proctoring can have, and that they ($n=3$) were personally discomforted with
the use of video and audio recording to monitor exams and using that to
actually identify cases of cheating. As P98 (School of Medicine & Health
Sciences; Respondus) describes

> The [proctoring software] utilizing the camera is an invasion of privacy,
> often didn’t work, and had the students so paranoid that they would email me
> to explain any movement they made. Plus, I realized that it would be
> difficult to ever prove anyone was actually cheating… I quit using the
> camera halfway through because of these problems.

Figure 1:  Most of the ($n=26$) instructors who used online proctoring tools
indicated they did not feel the proctoring tools they used were privacy
invasive (Q22). However, as demonstrated by a plurality of respondents
($n=12$; 46%), instructors were (slightly) more concerned about the privacy
risks the use of these tools posed to their students (Q24). Forth-six percent
($n=12$) of instructors at least somewhat agree that the tools offered a
reasonable tradeoff between student privacy and exam integrity (Q23). Most
($n=14$; 54%) participants felt the tools they used offered a good solution
for remotely monitoring exams (Q25).  Figure 2:  When asked if the use of the
tools makes it less likely that students will cheat Q19, 42% ($n=11$ of $26$)
at least somewhat agree. Figure 3:  When asked if the online exam proctoring
tool they used reported any potential cheating Q21, 38% ($n=10$ of $26$) said
that it had.

##### Effectiveness of Exam Proctoring

Educators who responded to the survey and indicated that they used online
proctoring tools ($n=26$) were asked about the effectiveness of these tools at
reducing cheating (Q19). Responses were mixed when considering if exam
proctoring tools reduced cheating. Eleven ($42\%$) educators who used online
proctoring tools either strongly agreed or somewhat agreed that exam
proctoring reduced cheating, while the same amount ($n=11$ of 26; 42%)
strongly or somewhat disagreed. Four ($15\%$) neither agreed nor disagreed.

There was less confidence that the proctoring software would actually catch
cheating (Q20). Roughly a third ($n=9$ of 26; 34%) of educators believed
proctoring tools would catch cheating at least 50% of the time. In contrast,
nearly two-thirds ($n=16$ of 26; 61.5%) believed it caught cheating up to 50%
of the time. Refer to Figure 2 for full details. Moreover, only 38% of
educators stated that the tools reported cases of cheating (Q21). See Figure 3
for more information.

This suggests that educators found the software to be more successful in
deterring cheating than in actually detecting or catching cheating. For
instance, educator P25 (School of Nursing, Proctorio) somewhat agreed that
exam proctoring reduced cheating but reported it catches cheating less than
25% of the time (Q20) and said, “They don’t prevent cheating - students can
look up ways on the Internet for workarounds. But they do deter cheating.”
Likewise, educator P125 (School of Medicine & Health Sciences; Respondus), who
somewhat disagreed that exam proctoring reduced cheating, stated it catches
cheating less than 25% of the time (Q20) and wrote, “The video monitor and
flagging is not great. Really doesn’t prevent cheating, may just deter for a
lot of people.”

Despite clearly different opinions on the effectiveness of exam proctoring
tools at preventing and identifying cheating, most of the educators ($n=14$ of
26; 54%) who used them reported that they either strongly or somewhat agreed
that they were a good solution in responses to Q25 (see Figure 1).

In response to Q12, with respect to drawbacks, three educators noted the
proctoring tools’ audio and video monitoring capabilities did not allow
instructors to fully inspect students’ exam conditions during assessments.
Eleven respondents indicated the tools did not eliminate cheating, and when
cheating was reported, three suggested that there were inconsistencies in the
reports, causing them frustration, and at least one false positive and one
false negative. Ultimately, this led two educators to suspect a subset of
students had likely cheated, but to have been unable to conclusively identify
which students had done so, despite their use of the proctoring tools.

(a) 2020/2021 academic year compared with Fall 2021.

(b) Fall 2021 compared with full return to in-person learning.

Figure 4:  Detailed visualization of how likely educators are to use remote
proctoring under circumstances similar to  (a) 2020/2021 academic year and
Fall 2021 (Q13 & Q15) and  (b) Fall 2021 and a full return to in-person
learning (Q15 & Q17).

##### Continued Use of Online Proctoring

We were interested in further exploring the impact of the COVID-19 pandemic on
the decision to use online exam proctoring tools. We asked the educators who
reported using online proctoring ($n=26$) if they would use online proctoring
tools again if they were teaching remotely under similar circumstances to
those of the 2020/21 academic year (Q13). Sixty-five percent ($n={17}$ of
${26}$) said they were likely to use online proctoring tools again under those
circumstances. While only 27% ($n={7}$ of ${26}$) said they were unlikely.

We followed up by asking why they would or would not use online exam
proctoring tools in such a situation Q14). Educators shared that it was either
the next best option ($n={3}$) or the only option ($n={3}$) when in-person
proctoring was not available. For instance, P47 (School of Nursing; Examsoft)
shared, “If we were unable to test in person, this would be our only option.”
For others ($n={2}$) it was to maintain exam integrity. As educator P25
(School of Nursing; Proctorio) highlighted, “It’s the only main way to control
for academic integrity when not face-to-face.” Educator P118 (School of
Nursing; Proctorio) reported pandemic safety was the reason and said, “If it
is about being safe during a pandemic, I will use remote proctoring software
every time.”

Next, we asked the same educators how likely would they be to use online
proctoring tools again if conditions were similar to Fall 2021 when in-person
learning resumed with masks and some hybrid options (Q15). Only 35% ($n={9}$
of ${26}$) reported they were likely to continue to use online proctoring
tools, while 46% ($n={12}$ of ${26}$; 54%) reported they were unlikely.

Finally, we asked the same educators how likely would they be to use online
proctoring tools again if they were teaching classes fully in-person without
hybrid options (Q17). We observed similar responses with 35% ($n={9}$ of
${26}$) reported they were likely to use the tools, and over half ($n={14}$ of
${26}$; 54%) reported they were unlikely. Detailed results can be found in
4(a) and 4(b).

We followed up again by asking why they would use online exam proctoring in
this situation. Some educators simply shared that there was no longer any need
to use online exam proctoring when in-person classes were taking place
($n={2}$), or that they preferred traditional in-person exam proctoring
($n={3}$). For example, educator P124 (College of Arts & Sciences; Respondus)
noted, “So if there is no concerns about pandemic, exams should definitely be
in person.” Others said they would still consider using online exam proctoring
for missed exams ($n={1}$) or other extenuating circumstances ($n={1}$), and
asynchronous quizzes($n={1}$). Some educators would continue to use online
exam proctoring for reasons of flexibility ($n={1}$), hybrid course offerings
($n={1}$), or because they prefer online exam proctoring ($n={1}$). Educator
P56 who referred to flexibility said, “I think it is a helpful option for
times when holding exams virtually provides flexibility for students and
faculty while still meeting course objectives and assessment standards.”

Our results suggest that many educators have used online exam proctoring as a
temporary expedient to manage assessments during pandemic induced remote
learning periods. However, our results also suggest that a subset of educators
will likely continue to use online proctoring as classes return to full in-
person learning.

###  4.2 RQ2: Privacy and Security Concerns

Figure 5:  When educators who had used online exam proctoring were asked to
report their level of concern about students installing software created by
online exam proctoring services on their personal computers Q27, more than
half ($n=14$ of $26$; 54%) reported that they were not at all concerned, while
27% ($n=7$ of $26$) were at least somewhat concerned.

##### Privacy Concerns

As with any application, there are possible privacy and security risks for
users. We asked educators responding to the survey questions regarding privacy
in relation to exam proctoring tools. One of the questions asked educators if
they thought monitoring tools were an invasion of privacy (Q22). Of the
educators that indicated using proctoring tools ($n={26}$ of ${125}$), the
majority either strongly disagreed ($n={6}$ of ${26}$) or somewhat disagreed
($n={10}$ of ${26}$) with the concept of a proctoring service being an
invasion of privacy. When the educators were separately asked if they were
specifically concerned about privacy risks to students (Q24), the majority
either somewhat agreed ($n={8}$ of ${26}$) or strongly agreed ($n={4}$ of
${26}$). See Figure 1 for full results.

When asked to elaborate about the specific factors that informed their views
about the privacy of online exam proctoring tools in an open response question
(Q26), one common theme ($n={9}$) among some of the educators who chose to use
online exam proctoring tools was a belief that the privacy risks was
acceptable. For instance, P28 (School of Medicine & Health Sciences;
Respondus) stated,

> Most digital tools have some level of privacy issues. Any program that can
> access an internal mic or video seems to be a privacy risk. There are other
> learning tools such as Voice thread that I believe have a privacy risk but
> see the benefit outweighing the risk and damage to the student.

In contrast, many educators ($n={9}$) expressed discomfort with online exam
proctoring tools. For example, educator P36 (School of Nursing; Examsoft)
stated,

> Any program that requires you to download a file, disables your system
> functionality, and automatically searches your computer for files to upload
> is a total invasion of privacy. Additionally, the proctoring service records
> the student at their most vulnerable- in their home environment where they
> sometimes forget they are being recorded. This leaves the potential for
> private matters being recorded and permanently on a server somewhere… if
> there is a data breach of this program, these videos could be out there for
> anyone to see.

Some participants ($n={3}$) called out the webcam as being privacy invasive,
like P125 (School of Medicine & Health Sciences; Respondus) who illustrated,
“Video monitoring is more intrusive.” Educator P58 (College of Arts &
Sciences; Respondus) stated that artificial intelligence used to monitor
student behavior in private was invasive when they said “AI required to
monitor and flag student behavior use of recordings in private setting.” And
educator P49 (College of Arts & Sciences; Respondus) added, “Glad to see the
tradeoff question: Yes, it is somewhat invasive but that is offset by exam
integrity.”

When asked whether they thought the remote exam proctoring tool they used
offered a reasonable tradeoff between student privacy and exam integrity
(Q23), respondents appeared to be hesitant to endorse the tools. Here, only a
plurality of participants ($n={12}$) indicated they agreed with the statement,
where only ($n={6}$) strongly agreed with the statement, and ($n={6}$)
somewhat agreed. See Figure 1 for full results.

##### Software Security Concerns

Exam proctoring services often require students to install specialized
software to enable proctoring. The required proctoring software is often in
the form of a browser extension that is added to students’ existing web
browsers or standalone software that must be installed on students’ personal
computers. As is the case with any custom software, there is risk of security
vulnerabilities.

We asked educators responding to the survey who had used remote proctoring
tools how concerned they were about students installing software created by
exam proctoring companies on their personal computers (Q27). Over half of
educators ($n={14}$ of ${26}$; 54%) were not at all concerned, while 31%
($n={8}$ of ${26}$) where slightly or somewhat concerned. Only 15% ($n={4}$ of
${26}$) were moderately or extremely concerned. Refer to Figure 5 for the full
results.

We then asked respondents to explain which factors led to their concern, or
lack of concern, regarding students installing exam proctoring software (Q27).
A number of educators ($n={8}$) voiced concerns about the software. Concerns
such as reliability issues ($n={3}$), potential invasion of student privacy
($n={2}$), security flaws ($n={1}$), negative impacts on computer
functionality ($n={1}$). Educator P36 (School of Nursing; Examsoft) share
concerns about privacy, “Again, any program that requires you to download a
file, disables your system functionality, and automatically searches your
computer for files to upload is a total invasion of privacy.” Whereas,
educator P124 (College of Arts & Sciences; Respondus) considered tradeoffs
between privacy and necessity when the educator said,

> No one likes to install software on their computer that could potentially be
> invasive. It’s necessary in this instance but I can see why someone would be
> reluctant to do so.

Still a number of educators ($n={6}$) did not have concerns, such as educator
P28 (School of Medicine & Health Sciences; Respondus) who noted, “We install
so much on our devices so I don’t see this as a higher risk than other
applications.” We also found statements describing a transfer of trust from
the institution, which licensed the software and made it available to
educators, to the exam proctoring software itself. This implied trust leads
educators to assume that the software has been through a vetting process. For
instance, educator P125 (School of Medicine & Health Sciences; Respondus)
responded, “If recommended by University then assume it is safe.” And educator
P25 (School of Nursing; Proctorio) added, “Just don’t know enough to answer
this question; defer to our [Online Learning and Instructional Technology]
team who vet the software.”

###  4.3 RQ3: Proctoring Methods

Figure 6:  Educators who reported using online exam proctoring tools ($n=26$
of $125$) were asked to select all monitoring methods they enabled (Q29). Over
80% of educators reported enabling the lockdown browser, and 50% of educators
enabled webcam recording during their online proctored exams. The educators
were also asked to select how comfortable they would feel about using each
monitoring type to monitor students during online proctored exams in their
course (Q33). Most educators were comfortable with a lockdown browser. A live
proctor not visible to students had the largest number of uncomfortable
educators, followed by eye movement tracking and web browser history
monitoring.

##### Enabling Monitoring Methods

Online exam proctoring services provide numerous types of student monitoring
methods. These monitoring techniques range from lockdown browsers that prevent
navigation to other sites during exam time, to more invasive monitoring that
may include webcams, screen sharing, the use of a live (human) proctor, and
even automated monitoring techniques such as eye tracking and network traffic
analysis. Educators must select the monitoring methods they deem appropriate
for proctoring students while they complete assessments.

We asked educators to report all of the monitoring methods they enabled in
their proctored exams (Q29). Most educators ($n={21}$ of ${26}$; 81%) reported
enabling the lockdown browser, which many educators find to be the least
invasive proctoring technique. Fifty percent of the educators ($n={13}$ of
${26}$) enabled webcam recording, and many educators ($n={11}$ of ${26}$; 42%)
enabled microphone recording and face detection. Still others ($n={8}$ of
${26}$; 31%) enabled the arguably more invasive monitoring methods of screen
recording and eye movement tracking. Refer to Figure 6 for the full results.

##### Monitoring Method Effectiveness

A majority ($n={16}$ of ${26}$; 62%) of educators reported that they would
enable the same monitoring methods again to administer another online
proctored exam (Q30). While 38% ($n={10}$ of ${26}$;) reported they would not
use ($n={3}$ of ${26}$; 11%), or were unsure if they would use ($n={7}$ of
${26}$; 27%), the same monitoring methods.

When asked what monitoring methods in their online proctored assessments they
would change and why (Q31). Educator P19 (College of Arts & Sciences;
Respondus), who wanted to remove the webcam monitoring said, “Students did not
feel comfortable or said they did not have a camera so we could not go that
route.” Furthermore, educator P124 (College of Arts & Sciences; Respondus) who
wanted to remove the face detection monitoring shared, “Facial detection not
as necessary I don’t turn on the option for Respondus to fire off warnings
when students face disappear from view, but I do watch the recordings later to
determine if there is any egregious violations.”

Additionally, educators reported technology issues with monitoring that relies
on the webcam or microphone. For instance, educator P47 (School of Nursing;
Examsoft) stated,

> There is no way of guaranteeing that the student’s webcam and microphone are
> working during the test. It is not until after that we can determine if they
> were working and by then, it’s too late.

Bandwidth and lack of staff to view the videos could also be an issue as
educator P103 (School of Business; Respondus) added, “Most students had
excuses not to have cameras, the low bandwidth was a problem with Respondus,
and we don’t have enough staff for watching/proctoring.”

##### Comfort With Monitoring Methods

We asked educators how comfortable they feel using each monitoring type during
an online proctored exam (Q33). Overall educators were comfortable with all 12
monitoring types presented to them. The lockdown browser monitoring method had
the largest number ($n={22}$ of ${26}$; 85%) of educators who reported being
comfortable, many $n={20}$ of ${26}$; 77%) of them extremely comfortable. This
is followed by internet activity monitoring at 65% ($n={17}$ of ${26}$) of
educators comfortable and keyboard restrictions at 54% ($n={14}$ of ${26}$).
Please refer to Figure 6 for the full results. These results are notably
inline with the results found by Balash et al. when they asked students to
select their comfort level with online exam proctoring monitoring methods[2].

Figure 7:  Educators who reported using online exam proctoring tools ($n=26$
of $125$) were asked to indicate how concerned they would be by students
sharing each type of information with exam proctoring companies (Q34). The
largest number of educators ($n=22$ of $26$; 85%) were concerned with students
sharing their social security number. While the fewest number of educators
($n=3$ of $26$; 12%) were concerned with students sharing their screen view.

##### Information Sharing Concern

We asked educators to report their level of concern on a 5-point Likert
concern scale for students sharing these various types of information with
exam proctoring companies (Q34). Educators were generally unconcerned with
most types of student information sharing, except for identifiable information
such as social security number, date of birth, street address and location
data. The student information that garnered the largest number ($n={22}$ of
${26}$; 85%) of concerned educators was social security number. Please refer
to Figure 7 for the full results.

##### Exploring Factors Influencing Proctor Usage

We performed exploratory analysis to determine potential factors that could
lead to increased/decreased usage of exam proctoring in the future. As we had
no priors and a small sample, we applied feature reduction analysis using
multiple logistic regressions, considering all possible factors and reducing
individual until the model either did not converge or the variance was no
longer improving. As this exploratory, we refrain from presenting odds ratios
and p-values in text and instead focus on factors that could be explored more
in future research. The factors we considered included educator comfort with
the 12 exam monitoring types (Q33), concern for 11 student information sharing
types (Q34), agreement that online exam proctoring tools makes it less likely
that students will cheat (Q19), the percentage of actual cheating found (Q20),
and if the tool reported potential cheating (Q21). As outcomes we considered
the likelihood of using online exam proctoring tools for assessments assuming
similar circumstances to the 2020/2021 academic year (Q13), Fall 2021 (Q15),
and a full return to in person learning (Q17). The outcomes were binned into
two levels, educators who were _Extremely Likely_ or _Somewhat Likely_ to use
proctoring and those who were not. The full models are found in Appendix B.

The leading factors that survived reduction were visible proctors, sharing
student info, and does proctoring actually prevent cheating. For instance, we
found (Table 3) a correlation with participants who were _Extremely
Comfortable_ or _Somewhat Comfortable_ with a live proctor not visible to
students and internet monitoring of students (Q33). Those participants were
more likely to use online exam proctoring tools given similar circumstances to
those of the 2020/2021 academic year (Q13). And we find a similar correlation
for those who _Strongly Agree_ or _Somewhat Agree_ that the use of online exam
proctoring tools makes it less likely that students will cheat (Q19). Future
work could design experiments around these factors to quantify the effects.

##  5 Discussion and Conclusions

We surveyed $(n=125)$ educators at a large academic institution about their
perceptions and use of online proctoring during the 2020/21 academic year. Of
those who responded to the survey, most ($n=99$; 79%) did not use online
proctoring, with some arguing that it was invasive or unnecessary.

Of those that did use online proctoring tools ($n=26$; 21%), many felt that
they did not have a choice either because of the necessity to maintain
academic integrity or because they were required to do so by their department.
Educators who used online proctoring were also not uniform in their view that
it actually helps to deter and detect cheating, despite noting that it was a
good solution under the circumstances. Furthermore, there was general comfort
among educators who used remote proctoring with the monitoring of students
(e.g., via video, screen share, or microphone); these educators were more
concerned with sharing student data (e.g., name, student identification, etc.)
with online proctoring companies.

Moving forward, even in a situation where there is full, in-person learning,
many educators that use online proctoring indicated they would continue to do
so, suggesting that more work is needed to address the potential privacy and
security risks for students and educators when using these tools.

##### Privacy Tradeoffs

There were marked differences between the educators who used online exam
proctoring and those who chose not to use the tools. Educators who did not use
online proctoring tools can generally be classified into one of two
categories: The first consists of educators who preferred to redesign their
assessments so they could be more easily completed remotely without concern
for academic integrity violations, such as open book/note/internet exams and
writing or project-based assignments. The second consist of those who
considered the tradeoffs between student privacy and the utility of the tools
and decided that the potential privacy and security risks to student test
takers outweighed the utility of the tools. Overall, the instructors who did
not use remote proctoring had the most thematically negative responses to
these tools and often highlighted the privacy risks and potential harms to
students.

Likewise, the educators in our study that did use online proctoring tools can
generally be classified into one of two categories: The first category are
those who were forced to use online proctoring services either by their
department, organizational unit, or as a standardized testing requirement.
Their opinions of the tools generally better matched those who chose not to
use them, but generally thought of them as less harmful and privacy invasive
than those who did not use online proctoring at all. The second category is
educators who considered the tradeoffs between student privacy and the utility
of the tools and decided that the need for academic integrity outweighed the
potential privacy and security risks to student test takers.

##### Educator Training and Guidance for Online Proctoring

Many educators in the study expressed a desire for training to better
understand the available proctoring tools and their impacts. Overall, they
demonstrated general knowledge about how exam proctoring tools function with
respect to monitoring students and how they restrict the use of unauthorized
resources. However, educators seemed unaware of the methods used to validate
students’ identities and what happens to students’ information after its
collection for this purpose.

This presents an opportunity for improved training and guidance at the
institutional level that provides the pros and cons about online proctoring
and associated privacy/security risks. Such training and guidance could also
include technical details on how exam proctoring tools can help educators
maintain principles of least monitoring by using the smallest number of
monitoring types necessary, given the constraints of the class. Moreover,
institutional involvement in such training could set clear recommendations to
set expectations for both educators and students.

##### Limitations on Enforcing Academic Integrity

Qualitative responses suggest that educators have broad skepticism of the
limitations about what remote proctoring tools can actually do to ensure exam
integrity. Many often highlighted the difference between the ability to deter
potential cheating versus the inability to detect motivated cheating. For
instance, online exam proctoring tools may prevent panic cheating, but they
will not necessarily stop or detect more planned or sophisticated cheating
techniques, such as secondary devices, virtual machines, or other workarounds.
Furthermore, even educators who use the tools are fairly split on whether
online proctoring actually deters and detects cheating.

##### Transfer of Trust

In the qualitative responses from educators, we find evidence of a transfer of
trust between the institutions who licence and provide the online exam
proctoring software and the software itself. A similar finding was reported by
Balash et al. [2] when surveying students on their opinions of online
proctoring, noting that students trusted the institution and since the
institution licensed online proctoring, they implicitly trusted online
proctoring tools. We found that the educators who used online proctoring
expressed the same sentiment. They believed that their institution would not
provide the software to proctor exams if it was not safe for students to
install on their computers.

Institutional support for third-party proctoring software, which conveys
credibility, makes the exam proctoring software appear safer and less
potentially problematic because educators assume that institutions have
properly vetted the software and the methods used by the proctoring services.

It is unclear that such trust is warranted. All software has inherent risks of
security vulnerabilities, and recent major security and privacy incidents have
shown that online exam proctoring software has been subject to both major data
breaches and to security vulnerabilities that allowed remote activation. Given
the capabilities of exam proctoring software to monitor users and disable
system functionality, extra precautions should be taken to reduce security
risks to students who are required to install the software.

##### Implications to Security and Privacy

Remote proctoring systems are naturally invasive. When operating correctly,
they monitor and restrict how students can interact with their own computers.
The consequences of security vulnerabilities and breaches (cf. [1, 29, 3, 27])
are significant, especially given the private information collected about
students—for example, their physical locations, photos and videos of their
environment, and information about their computing devices.

Understanding how and why users choose (or are forced to use) software that
could harm their privacy and security is a critical research need. This paper
examines the perceptions of the decision-makers who choose whether or not to
require remote proctoring—a form of monitoring software that has seen
explosive growth. As argued above, providing additional guidance and training
to educators, and heightening their awareness of the privacy and security
risks that these systems impose on their students, is paramount. More
generally, as with other technologies that aim at restricting and monitoring
user functionality (e.g., remote IT management software), an argument can be
made that limiting users’ ability to control their own devices is antithetical
to security and privacy. Given the proliferation of remote proctoring, there
is an urgent need to better understand not only the potential for abuse and
misuse of these systems, but also the perceptions of the educators who have
the power to decide if and how they are used.

#### Data and Source Availability

All data, scripts, and qualitative codebooks are available at the following
repository: https://github.com/gwusec/2023-USENIX-Educator-Perspectives-of-
Exam-Proctoring

#### Acknowledgements

We thank the anonymous shepherd and reviewers for improving this manuscript
and preparing it for publication. This material is based upon work supported
by the National Science Foundation under Grant Nos. 1845300, 2138654, and
2138078.

## References

  * [1] Lawrence Abrams.  Proctoru confirms data breach after database leaked online.  https://www.bleepingcomputer.com/news/security/proctoru-confirms-data-breach-after-database-leaked-online/.
  * [2] David G. Balash, Dongkun Kim, Darika Shaibekova, Rahel A. Fainchtein, Micah Sherr, and Adam J. Aviv.  Examining the examiners: Students’ privacy and security perceptions of online proctoring services.  In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021), pages 633–652. USENIX Association, August 2021.
  * [3] Ben Burgess, Avi Ginsberg, Edward W. Felten, and Shaanan Cohney.  Watching the watchers: bias and vulnerability in remote proctoring software.  In 31st USENIX Security Symposium (USENIX Security 22), pages 571–588, Boston, MA, August 2022. USENIX Association.
  * [4] Lilah Burke.  Cutting the in-person semester short.  Inside Higher Ed. https://www.insidehighered.com/news/2020/11/17/colleges-end-person-instruction-early-due-covid-19-spread, November 2020.
  * [5] Gemma Cherry, Michael O’Leary, Oksana Naumenko, Li ann Kuan, and Linda Waters.  Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ?  Computers and Education Open, 2021.
  * [6] Simon Coghlan, Tim Miller, and Jeannie Paterson.  Good proctor or “Big Brother”? AI Ethics and Online Exam Supervision Technologies.  arXiv preprint arXiv:2011.07647, 2020.
  * [7] Shaanan Cohney, Ross Teixeira, Anne Kohlbrenner, Arvind Narayanan, Mihir Kshirsagar, Yan Shvartzshnaider, and Madelyn Sanfilippo.  Virtual classrooms and real harms: Remote learning at U.S. universities.  In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021), pages 653–674. USENIX Association, August 2021.
  * [8] Lee William Daffin Jr and Ashley A Jones.  Comparing Student Performance on Proctored and Non-Proctored Exams in Online Psychology Courses.  Online Learning, 22(1):131–145, 2018.
  * [9] Ann Davis, Richard S. Rand, and Robert A. Seay.  Remote proctoring: The effect of proctoring on grades.  2016\.
  * [10] Ahu Genis-Gruber and Gerold Weisz.  Challenges of online exam systems in the covid-19 pandemic era.  Measurement Methodologies to Assess the Effectiveness of Global Online Learning, 2022.
  * [11] Patricia A Goedl and Ganesh B Malla.  A Study of Grade Equivalency between Proctored and Unproctored Exams in Distance Education.  American Journal of Distance Education, pages 1–10, 2020.
  * [12] Susan Grajek.  EDUCAUSE COVID-19 QuickPoll Results: Grading and Proctoring.  EDUCAUSE Research Notes, April 2020.
  * [13] Therese C Grijalva, Joe Kerkvliet, and Clifford Nowell.  Academic Honesty and Online Courses.  College Student Journal, 40(1), 2006.
  * [14] Helen C. Harton, Salomi Aladia, and Anita Gordon.  Faculty and student perceptions of cheating in online vs. traditional classes.  Online Journal of Distance Learning Administration, 22(4), 2019\.
  * [15] Drew Harwell.  Mass School Closures in the Wake of the Coronavirus are Driving a New Wave of Student Surveillance.  The Washington Post. https://www.washingtonpost.com/technology/2020/04/01/online-proctoring-college-exams-coronavirus/, April 2020.
  * [16] Cristyne Hébert.  Online remote proctoring software in the neoliberal institution: Measurement, accountability, and testing culture.  in education, 2021.
  * [17] Shawn Hubler.  Keeping Online Testing Honest? Or an Orwellian Overreach?  The New York Times. https://www.nytimes.com/2020/05/10/us/online-testing-cheating-universities-coronavirus.html, May 2020.
  * [18] Kenrie Hylton, Yair Levy, and Laurie P Dringus.  Utilizing webcam-based proctoring to deter misconduct in online exams.  Computers & Education, 92:53–63, 2016.
  * [19] Faten F Kharbat and Ajayeb S Abu Daabes.  E-proctored Exams During the COVID-19 Pandemic: A Close Understanding.  Education and Information Technologies, pages 1–17, 2021.
  * [20] Fleur L. Meulmeester, Eline A. Dubois, C. (Tineke) Krommenhoek van Es, Peter G. M. de Jong, and Alexandra M J Langers.  Medical students’ perspectives on online proctoring during remote digital progress test.  Medical Science Educator, pages 1 – 5, 2021.
  * [21] Sandra Gudiño Paredes, Felipe de Jesús Jasso Peña, and Juana María de La Fuente Alcazar.  Remote proctored exams: Integrity assurance in online education?  Distance Education, 42(2):200–218, 2021.
  * [22] Anushka Patil and Jonah Engel Bromwich.  How It Feels When Software Watches You Take Tests.  The New York Times. https://www.nytimes.com/2020/09/29/style/testing-schools-proctorio.html, September 2020.
  * [23] Joseph A. Rios and Ou Lydia Liu.  Online Proctored Versus Unproctored Low-Stakes Internet Test Administration: Is There Differential Test-Taking Behavior and Performance?  American Journal of Distance Education, 31(4):226–241, 2017.
  * [24] Madeleine Schultz and Damien L. Callahan.  Perils and promise of online exams.  Nature Reviews. Chemistry, 6:299 – 300, 2022.
  * [25] Dendir Seife and Maxwell R. Stockton.  Cheating in online courses: Evidence from online proctoring.  Computers in Human Behavior Reports, 2:100033, 2020.
  * [26] Sarah Kozel Silverman, Autumm Caines, Christopher Casey, Belen Garcia de Hurtado, Jessica L. Riviere, Alfonso Sintjago, and Carla Vecchiola.  What happens when you close the door on remote proctoring? moving toward authentic assessments with a people-centered approach.  To Improve the Academy, 2021.
  * [27] Taylor Swaak.  A vulnerability in proctoring software should worry colleges, experts say.  https://www.chronicle.com/article/a-vulnerability-in-proctoring-software-should-worry-colleges-experts-say.
  * [28] Shea Swauger.  Software that Monitors Students During Tests Perpetuates Inequality and Violates their Privacy.  MIT Technology Review, August 2020.
  * [29] Daniël Verlaan.  Omstreden software studenten blijkt onveilig: hackers konden meegluren.  https://www.rtlnieuws.nl/nieuws/nederland/artikel/5273869/studenten-nederland-proctorio-hacken-plugin-uva.
  * [30] George R Watson and James Sottile.  Cheating in the Digital Age: Do Students Cheat More in Online Courses?  In Society for Information Technology & Teacher Education International Conference. Association for the Advancement of Computing in Education (AACE), 2008.
  * [31] Daniel Woldeab and Thomas Brothen.  21st Century Assessment: Online Proctoring, Test Anxiety, and Student Performance.  International Journal of E-Learning & Distance Education, 34(1), 2019.
  * [32] Rattaphon Wuthisatian.  Student exam performance in different proctored environments: Evidence from an online economics course.  International Review of Economics Education, 35, 2020.

##  Appendix A Survey Instrument

  1. Q1

Are you either a full-time faculty or part-time/adjunct faculty member at GW?

     * $\bigcirc$

Yes

     * $\bigcirc$

No

  2. Q2

What is the primary organization unit or school you are associated with at GW?

     * $\bigcirc$

[_List of 14 schools or organizational units_]

     * $\bigcirc$

Prefer Not To Disclose

     * $\bigcirc$

Other (Please Specify)

  3. Q3

What subject area(s) (e.g., Math, English) did you teach during the 2020/2021
academic year? Enter “N/A” if you prefer not to disclose this information.

  4. Q4

These are the major online exam proctoring tools available at GW: For each
tool note whether you were aware of its availability during the 2020/2021
Academic Year.  Aware Unaware I’m Unsure Examity $\bigcirc$ $\bigcirc$
$\bigcirc$ Examsoft $\bigcirc$ $\bigcirc$ $\bigcirc$ Honorlock $\bigcirc$
$\bigcirc$ $\bigcirc$ Proctor Track (Verificent) $\bigcirc$ $\bigcirc$
$\bigcirc$ ProctorU $\bigcirc$ $\bigcirc$ $\bigcirc$ Question mark $\bigcirc$
$\bigcirc$ $\bigcirc$ Respondus $\bigcirc$ $\bigcirc$ $\bigcirc$ Software
Secure PSI (Remote Proctor Now) $\bigcirc$ $\bigcirc$ $\bigcirc$

  5. Q5

To the best of your understanding, how do online exam proctoring tools work?

  6. Q6

Did you use any online exam proctoring tools to assist in administering any
exams or assignments during the Fall 2020/Spring 2021 academic year?

     * $\bigcirc$

Yes

     * $\bigcirc$

No

     * $\bigcirc$

I’m Unsure

[Q6N1-Q6N5 only if Q6 is No or I’m Unsure]

    1. Q6N1

Did you consider using any online exam proc. tools?

       * $\bigcirc$

Yes

       * $\bigcirc$

No

       * $\bigcirc$

I’m Unsure

    2. Q6N2

Why did you consider using online exam proc. tools?

    3. Q6N3

Why did you ultimately choose not to use online exam proctoring tools?

    4. Q6N4

For course assessments, like exams, quizzes, etc., how did you manage student
academic integrity for online classes during the 2020/2021 academic year?
Please explain your decision making process

    5. Q6N5

If you were teaching remotely under similar circumstances to those of the
2020/2021 academic year, and you decided to administer an exam online, how
likely would you be to use online exam proctoring tools?

       * $\bigcirc$

[_Five-point Likert likelihood scale_]

[Q7-Q34 are shown only if answer to Q6 is Yes]

  7. Q7

Which online proctoring service did you use most recently?

     * $\bigcirc$

[_List of online proctoring services from Q4_]

  8. Q8

Which factors did you consider when making your decision to use online
proctoring services?

  9. Q9

What kind of assessment did you administer most recently using an online exam
proctoring service?

     * $\bigcirc$

Course quiz

     * $\bigcirc$

Course exam (e.g. test, midterm exam, final exam)

     * $\bigcirc$

I have not administered an exam with online proc.

     * $\bigcirc$

Other (Please Specify)

  10. Q10

How many online proctored assessments have you administered?

     * $\bigcirc$

0

     * $\bigcirc$

1

     * $\bigcirc$

2

     * $\bigcirc$

3

     * $\bigcirc$

4

     * $\bigcirc$

5+

  11. Q11

In your experience, what were the main benefits of using online exam
proctoring?

  12. Q12

In your experience, what are the main drawbacks of using online exam
proctoring?

  13. Q13

If you were teaching remotely under similar circumstances to those of the
2020/2021 academic year, and you decided to administer an exam online, how
likely would you be to use online exam proctoring tools?

     * $\bigcirc$

[_Five-point Likert likelihood scale_]

  14. Q14

Why would or wouldn’t you consider using online exam proctoring?

  15. Q15

Based on your prior experience with online exam proctoring, in the current
teaching environment at GW during Fall 2021, how likely are you to use online
exam proctoring tools for these online assessments?

     * $\bigcirc$

[_Five-point Likert likelihood scale_]

  16. Q16

Why are, or aren’t you considering administering assessment online during Fall
2021?

  17. Q17

Based on your prior experience with online exam proctoring, assuming the end
of the pandemic and a full return to in person learning, how likely would you
be to use online exam proctoring tools for these assessments?

     * $\bigcirc$

[_Five-point Likert likelihood scale_]

  18. Q18

Why would, or wouldn’t you consider using online exam proctoring if in person
learning resumed?

##### Understanding of Functions

In this part of the survey you will be asked about the functionality of online
exam proctoring services.

  19. Q19

The use of online exam proctoring tools makes it less likely that my students
will cheat on an exam.

     * $\bigcirc$

[_Five-point Likert agreement scale_]

  20. Q20

I believe that online proctoring will catch cheating this percent of the time
when students are actually cheating:

     * $\bigcirc$

0-25%

     * $\bigcirc$

26-50%

     * $\bigcirc$

51-75%

     * $\bigcirc$

76-100%

     * $\bigcirc$

Prefer not to answer

  21. Q21

Did [Exam Proctoring Tool] report any potential cheating during an exam?

     * $\bigcirc$

No

     * $\bigcirc$

Yes

     * $\bigcirc$

I’m Unsure

     * $\bigcirc$

Prefer not to answer

You indicated above that you used [Exam Proctoring Tool] recently to
administer an online exam. Please indicate your level of agreement with the
following statements based on your experience with [Exam Proctoring Tool].

  22. Q22

I think [Exam Proctoring Tool] is privacy invasive.

     * $\bigcirc$

[_Five-point Likert agreement scale_]

  23. Q23

I think [Exam Proctoring Tool] offers a reasonable tradeoff between student
privacy and the integrity of the exam.

     * $\bigcirc$

[_Five-point Likert agreement scale_]

  24. Q24

I am concerned about the risks to student privacy by using [Exam Proctoring
Tool].

     * $\bigcirc$

[_Five-point Likert agreement scale_]

  25. Q25

I think [Exam Proctoring Tool] is a good solution for monitoring remote
examinations.

     * $\bigcirc$

[_Five-point Likert agreement scale_]

  26. Q26

Which specific factors inform your views about the privacy of [Exam Proctoring
Tool]?

  27. Q27

How concerned are you about students installing software created by [Exam
Proctoring Tool] on their personal computers?

     * $\bigcirc$

[_Five-point Likert concerned scale_]

  28. Q28

Which factors led to your concern, or lack of concern, about students
installing [Exam Proctoring Tool] software?

##### Methods

In this part of the survey you will be asked about the methods employed by
online exam proctoring services.

Previously, you indicated that you administered an online exam via [Exam
Proctoring Tool] offered at GW. Please refer to that experience in answering
the following questions.

  29. Q29

Select all monitoring methods you enabled in your proctored exams. [Select all
that apply]

[_Refer to Figure 6 for the list of monitoring methods_]

  30. Q30

If you were to administer another online proctored exam would you enable the
same monitoring options again?

     * $\bigcirc$

No

     * $\bigcirc$

Yes

     * $\bigcirc$

I’m Unsure

  31. Q31

You indicated that you used the following monitoring methods in your online
proctored assessments: [Monitoring Methods] Please explain why you felt these
monitoring options worked well, or why they would work well proctoring a
future exam.

  32. Q32

What monitoring methods in your online proctored assessments would change and
why?

  33. Q33

For each exam monitoring type please select how comfortable you would feel
about using it to monitor students during online proctored exams in your
course.

[_Five-point Likert comfort scale_]

[_List of monitoring types from Q29_]

##### Privacy Concerns

In this part of the survey you will be asked about the benefits and potential
risks you associate with online exam proctoring.

  34. Q34

Many online proctoring tools require students to share information with the
exam proctoring companies. For each type of information below indicate whether
you would be concerned by students sharing this information with exam
proctoring companies.

[_Participants selected from a five-point Likert concern scale from "Not at
all Concerned" to "Extremely Concerned" for each information type_]

[_Refer to Figure 7 for the list of information types_]

##### End of Survey

We thank you for your time spent taking this survey. Your response has been
recorded.

##  Appendix B Additional Figures and Tables

Table 3:  Binomial logistic regression model to describe which factors
influenced the likelihood of using online exam proctoring tools for
assessments assuming similar circumstances to those of the 2020/2021 academic
year (_Extremely likely_ or _Somewhat likely_ responses to question Q13). The
Aldrich-Nelson pseudo $R^{2}$ of the model is 0.65.

Factor | Estimate | Odds ratio | Error | z value | Pr(>|z|) |
---|---|---|---|---|---|---
(Intercept) | -1.31 | 0.27 | 0.99 | -1.33 | 0.19 |
$\text{Live Proctor Visible}\in\\{\textit{Ext Comf., Some Comf.}\\}$ | -2.21 | 0.11 | 1.90 | -1.16 | 0.25 |
$\text{Live Proctor Not Visible}\in\\{\textit{Ext Comf., Some Comf.}\\}$ | 4.21 | 67.20 | 2.12 | 1.99 | 0.05 | *
$\text{Internet Monitoring}\in\\{\textit{Ext Comf., Some. Comf.}\\}$ | 3.77 | 43.20 | 2.24 | 1.68 | 0.09 | .
$\text{Webcam Recording}\in\\{\textit{Ext Comf., Some. Comf.}\\}$ | -3.48 | 0.03 | 2.24 | -1.56 | 0.12 |
$\text{Student Street Address}\in\\{\textit{Slight Conc., Not Conc.}\\}$ | 1.63 | 5.10 | 1.54 | 1.06 | 0.29 |
$\text{Less Cheating}\in\\{\textit{Str Agree, Some Agree}\\}$ | 2.41 | 11.20 | 1.37 | 1.76 | 0.08 | .

Signif. codes: $\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt***\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}}\widehat{=}<0.001$;
$\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt**\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}
}\widehat{=}<0.01$;$\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt*\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}
}\widehat{=}<0.05$; $\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt.\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}
}\widehat{=}<0.1$

Table 4:  Binomial logistic regression model to describe which factors
influenced the likelihood of using online exam proctoring tools for
assessments assuming a similar teaching environment to Fall 2021 (_Extremely
likely_ or _Somewhat likely_ responses to question Q15). The Aldrich-Nelson
pseudo $R^{2}$ of the model is 0.74.

Factor | Estimate | Odds ratio | Error | z value | Pr(>|z|) |
---|---|---|---|---|---|---
(Intercept) | -3.09 | 0.01 | 1.68 | -1.84 | 0.07 | .
$\text{Live Proctor Visible}\in\\{\textit{Ext Comf., Some Comf.}\\}$ | -1.90 | 0.15 | 1.67 | -1.13 | 0.26 |
$\text{Live Proctor Not Visible}\in\\{\textit{Ext Comf., Some Comf.}\\}$ | -3.17 | <0.01 | 2.42 | -1.31 | 0.19 |
$\text{Web Brwoser History}\in\\{\textit{Ext Comf., Some Comf.}\\}$ | 2.71 | 15.00 | 1.57 | 1.73 | 0.08 | .
$\text{Face Detection}\in\\{\textit{Ext Comf., Some Comf.}\\}$ | 1.69 | 5.40 | 2.71 | 0.62 | 0.53 |
$\text{Mouse Movement Tracking}\in\\{\textit{Ext Comf., Some Comf.}\\}$ | -4.54 | <0.01 | 2.91 | -1.56 | 0.12 |
$\text{Student Room Scan}\in\\{\textit{Slight Conc., Not Conc.}\\}$ | 5.31 | 203.00 | 2.44 | 2.18 | 0.03 | *

Signif. codes: $\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt***\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}}\widehat{=}<0.001$;
$\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt**\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}
}\widehat{=}<0.01$;$\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt*\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}
}\widehat{=}<0.05$; $\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt.\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}
}\widehat{=}<0.1$

Table 5:  Binomial logistic regression model to describe which factors
influenced the likelihood of using online exam proctoring tools for
assessments assuming the end of the pandemic and a full return to in person
learning (_Extremely likely_ or _Somewhat likely_ responses to question Q17).
The Aldrich-Nelson pseudo $R^{2}$ of the model is 0.72.

Factor | Estimate | Odds ratio | Error | z value | Pr(>|z|) |
---|---|---|---|---|---|---
(Intercept) | -3.07 | 0.05 | 1.99 | -1.54 | 0.12 |
$\text{Live Proctor Visible}\in\\{\textit{Ext Comf., Some Comf.}\\}$ | -4.09 | 0.02 | 1.86 | -2.19 | 0.03 | *
$\text{Eye Movement Tracking}\in\\{\textit{Ext Comf., Some Comf.}\\}$ | -1.84 | 0.16 | 2.60 | -0.70 | 0.48 |
$\text{Face Detection}\in\\{\textit{Ext Comf., Some Comf.}\\}$ | 6.32 | 557.00 | 3.25 | 1.95 | 0.05 | .
$\text{Mouse Movement Tracking}\in\\{\textit{Ext Comf., Some Comf.}\\}$ | -6.00 | <0.01 | 3.29 | -1.82 | 0.07 | .
$\text{Internet Monitoring}\in\\{\textit{Ext Comf., Some. Comf.}\\}$ | 4.47 | 87.00 | 2.71 | 1.65 | 0.10 | .
$\text{Student Full Name}\in\\{\textit{Slight Conc., Not Conc.}\\}$ | 3.98 | 53.40 | 1.97 | 2.02 | 0.04 | *
$\text{Student Photo ID}\in\\{\textit{Slight Conc., Not Conc.}\\}$ | -2.29 | 0.10 | 2.01 | -1.14 | 0.26 |

Signif. codes: $\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt***\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}}\widehat{=}<0.001$;
$\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt**\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}
}\widehat{=}<0.01$;$\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt*\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}
}\widehat{=}<0.05$; $\text{\leavevmode\ltxml@iqmark@open\textquoteleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt.\textquoteright\ltxml<EMAIL_ADDRESS>0.0002pt{}
}\widehat{=}<0.1$

Figure 8:  Educators who reported using online exam proctoring tools ($n=26$
of $125$) were asked to indicate how concerned they would be by students
sharing each type of information with exam proctoring companies (Q34). The
concern for each type of information is shown by the specific exam proctoring
tool that was used. The largest number of educators ($n=22$ of $26$; 85%) were
concerned with students sharing their social security number. While the fewest
number of educators ($n=3$ of $26$; 12%) were concerned with students sharing
their screen view.

##  Appendix C Qualitative Codebook

  * •

monitor-students (65)

_webcam (31), track-behavior (18), record-session (12), track-eye-movement
(8), track-environment (4), prevent-cheating (3), verify-identity (3),
keystrokes (2), flag-anomalies (2), log-activities (2), track-location (1),
tracks-IP (1), microphone (1)_

  * •

do-not-know (32)

_unaware (6)_

  * •

restrict-activities (30)

_lockdown-browser (12), prevent-cheating (1)_

  * •

harmful-to-students (24)

_invasive (7), intrusive (3), increased-stress (3), false-positives (2),
increased-anxiety (1), decrease-learning (1), confusion (1), fairness-concerns
(1), fear (1)_

  * •

timed-tests (21)

_one-q-at-a-time (2)_

  * •

flag-anomalies (19)

  * •

unaware (19)

_did-not-pay-attention (1)_

  * •

written-papers (17)

  * •

open-book-exam (14)

  * •

NA (12)

  * •

in-person-exams (11)

  * •

trusts-students (11)

  * •

privacy-concerns (11)

  * •

open-book (11)

  * •

does-not-eliminate-cheating (11)

_students-cheated (2), prohibited-materials (2), second-computer (1),
workarounds (1), smart-phones (1), physical-notes (1)_

  * •

not-consider (11)

_prefer-in-person (3), cheating-too-easy (3), no-need (2), student-privacy
(1), more-equity-in-person (1)_

  * •

stop-cheating (10)

_use-web-resources (5), device-use (1)_

  * •

cheating (10)

_caught-students (3), reduce (2), reduce-cheating (1)_

  * •

do-not-use (10)

  * •

possible-exam-method (9)

_convenient (1), only-option (1), used-previously (1), preferred-online (1),
midterm (1), sub-in-person-proctor (1)_

  * •

no-exams (9)

  * •

acceptable (9)

_benefit-outweighs-risk (3), if-lockdown-only (2), awkward-but-necessary (1),
request-video-deletion (1), monitored-in-person (1), some-privacy-issues (1),
used-as-student (1)_

  * •

honor-code (9)

  * •

discomfort (9)

_remote-data-storage (3), recorded-when-vulnerable (2), not-worth-risks (2),
searches-files (1), ineffective-tool (1), assume-student-guilt (1), controls-
computer (1)_

  * •

does-not-prevent-cheating (9)

  * •

unchanged (9)

_deters-cheating (4), resource-restrictions (4), webcam (3), lockdown-browser
(3), methods-provide-evidence-of-student-activity (1), zoom-proctoring (1),
eye-movement-tracking (1)_

  * •

concern (8)

_reliability-issues (3), invasion-of-privacy (2), connectivity-issues (1),
forcing-students-to-use (1), requires-download (1), ways-to-get-around-
protections (1), searches-computer-for-files (1), disables-system-
functionality (1), security-software (1)_

  * •

unsure-how-to-use (8)

  * •

application-based-questions (8)

  * •

projects (7)

  * •

blackboard (7)

_open-book (2), assignments (1), proctored-exam (1), safe-assign (1)_

  * •

technology-glitches (7)

_camera (2), program-working (1), causes-stress-for-students (1), VPNs (1),
old-device (1), computer-freezes (1), microphone (1), poor-internet-connection
(1), causes-stress-for-instructor (1)_

  * •

only-option-available (7)

_on-blackboard (3), program-decided-to-standardize (1), proctorio (1)_

  * •

unsure (7)

_what-happens-to-collected-data (1), defer-to-admin (1), of-installation-
process (1), of-any-malware (1)_

  * •

would-consider (6)

_prefer-online-proctoring (1), for-quizzes (1), in-hybrid-courses (1), offers-
flexibility (1), ensure-test-validity (1)_

  * •

written-based-exams (6)

  * •

student-privacy (6)

_third-party-access-to-data (1), camera (1), monitoring (1)_

  * •

already-using-online-proctoring (6)

_department-using (2), RPNow (1), examplify (1), proctorio (1), nursing-
students-standardized-exams (1), Respondus-Monitor (1)_

  * •

required-to-use (6)

_department (3)_

  * •

lack-of-concern (6)

_it-worked-for-most-students (1), only-used-when-accessing-the-exam (1), have-
read-privacy-policy (1), not-higher-risk-than-other-applications (1),
recommended-by-university (1), widespread-use (1)_

  * •

continue-remote-proctoring (5)

_final-exam-date-late (1)_

  * •

trust-in-students (5)

  * •

safe-assign (5)

  * •

project-based-exams (5)

  * •

does-not-trust-tools (5)

_langauge-exam (1)_

  * •

ease-of-use (5)

_dont-have-to-write-open-book-exam (2), delivery (1), setup (1), flexible-
timing (1)_

  * •

no-privacy-issue (4)

  * •

did-not-have-time (4)

  * •

may-consider (4)

_depends-on-students (1), asynch-quizzes (1), extenuating-circumstances (1),
for-missed-exams (1)_

  * •

tests-worth-little (4)

  * •

sign-integrity-code (4)

  * •

none (4)

  * •

no-assessments (4)

  * •

difficult-to-cheat (4)

  * •

proctored-exam (4)

_by-a-friend (1)_

  * •

take-home-exams (4)

  * •

covid-safety (4)

  * •

zoom (4)

_video-on (3), mic-on (1), polls (1)_

  * •

prefers-in-person (4)

  * •

unconcerned (3)

  * •

test-versions (3)

  * •

tools-vary (3)

  * •

inconsistent-proctor-reports (3)

_reports-of-videos (2)_

  * •

control-computer (3)

_restrict-activities (2)_

  * •

test-integrity (3)

  * •

next-best-option (3)

  * •

open-note (3)

  * •

only-option-remotely (3)

  * •

required (3)

  * •

ensures-fairness (3)

  * •

enforce-exam-rules (3)

  * •

concern-with-camera (3)

  * •

technology-issues (3)

_recording (2), low-bandwidth (1), no-verifying-technology-works-before-exam
(1)_

  * •

checked-individual-assignments (3)

  * •

essay-based-exam (3)

  * •

covid (3)

_absence (1), infection-risk (1)_

  * •

oral-exams (3)

  * •

badly (3)

  * •

unclear (3)

  * •

same-as-previous (3)

  * •

does-not-use (3)

  * •

recommended (3)

_respondus-lockdown-browser (1), zoom-room (1), by-school (1)_

  * •

limits-academic-dishonesty (2)

  * •

academic-integrity (2)

_test-integrity (1)_

  * •

remove-monitoring (2)

_webcam (1), face-detection (1)_

  * •

no-benefits (2)

  * •

easy-to-implement (2)

  * •

saves-faculty-time (2)

  * •

implies-lack-of-trust (2)

  * •

system-is-limited (2)

  * •

outsource-proctoring (2)

  * •

would-not-use-any-monitoring-methods (2)

  * •

exam-integrity (2)

  * •

not-required (2)

  * •

self-proctoring (2)

_zoom (1), can-not-identify-sophisticated-cheating (1)_

  * •

used-previously (2)

  * •

same-as-before (2)

  * •

used-default-methods (2)

  * •

student-discomfort (2)

_monitoring (1)_

  * •

unethical (2)

_malware (1)_

  * •

online-courses (2)

  * •

unreliable (2)

  * •

used-more-space (2)

  * •

security (2)

  * •

student-accessibility (2)

_what-worked-on-their-computer (1)_

  * •

fairness (2)

  * •

cost (2)

  * •

ensures-integrity (2)

  * •

convenience (1)

  * •

students-select-enviroment (1)

  * •

professor-not-in-control (1)

  * •

easily-available (1)

_business-school (1)_

  * •

wants-better-info-from-university (1)

  * •

questions-during-exam-difficult (1)

  * •

open-internet (1)

  * •

verify-student-id (1)

  * •

no-answer-keys (1)

  * •

avoids-technology-problems (1)

_providing-links-for-browser-downloads (1), practice-quizzes (1)_

  * •

not-feasible (1)

  * •

control-testing-environment (1)

  * •

monitor-computer (1)

_computer-inputs (1), other-software (1)_

  * •

would-use-recording (1)

  * •

completion-grading (1)

  * •

stay-on-schedule (1)

  * •

tradeoff (1)

_equity-versus-privacy (1), additional-stress (1)_

  * •

clinicals (1)

  * •

popularity (1)

  * •

self-proctored (1)

  * •

used-lockdown (1)

  * •

accountability (1)

  * •

portfolio-show (1)

  * •

exam-type (1)

  * •

lax-on-cheating (1)

  * •

lack-of-trust (1)

  * •

exam-seriousness (1)

  * •

exam-questions-posted-online (1)

_chegg (1)_

  * •

cheating-did-not-help (1)

  * •

add-monitoring (1)

_copy-paste-restrictions (1)_

  * •

lack-honesty (1)

  * •

would-not-use-video (1)

  * •

higher-level-of-proctoring (1)

  * •

limit-student-stress (1)

  * •

can-be-avoided (1)

  * •

ease-of-grading (1)

  * •

students-designed-rubric (1)

  * •

stability (1)

  * •

colleagues-used-it (1)

_respondus (1)_

  * •

wants-feedback-from-recorded-sessions (1)

  * •

uncomfortable (1)

_used-specifically-for-cheating (1)_

  * •

department-requires (1)

  * •

harder-tests (1)

  * •

no-answer (1)

  * •

could-not-use-camera (1)

_students-uncomfortable (1)_

  * •

false-positives (1)

  * •

AI-magic (1)

  * •

standerized-tests (1)

  * •

used-what-they-could (1)

  * •

can-be-done-online (1)

  * •

test-fairness (1)

  * •

SON-policy (1)

  * •

well (1)

  * •

written-language-proficiency (1)

  * •

provide-record-of-student-behavior (1)

  * •

would-modify-exam (1)

_does-not-need-proctoring (1)_

  * •

test-in-class (1)

_student-preference (1)_

  * •

RP-exams (1)

  * •

university-should-train-faculty (1)

  * •

grade-confidence (1)

  * •

simulate-in-class-exam (1)

  * •

async-exams (1)

  * •

inaccurate-reporting (1)

_caused-lockdown (1)_

  * •

one-on-one-exam (1)

  * •

asks-students (1)

  * •

poor-integration-blackboard (1)

  * •

harms-learning (1)

  * •

special-arrangements (1)

  * •

enforce-time-limit (1)

  * •

would-use-lockdown (1)

  * •

not-applicable (1)

  * •

familiarity (1)

  * •

methods-not-necessary (1)

  * •

test-efficiency (1)

  * •

students-should-obey-us (1)

  * •

remain-on-schedule (1)

  * •

presentations (1)

  * •

prefer-monitor-in-person (1)

  * •

smaller-assignments (1)

  * •

not-allowed (1)

  * •

off-site-exam (1)

  * •

login-process (1)

  * •

performance-based-exam (1)

  * •

is-necessary-tradeoff (1)

  * •

proctored-exam-virtually (1)

  * •

uncomfortable-watching-recordings (1)

_student-living-situation (1)_

  * •

no-choice-of-methods (1)

  * •

would-keep-settings (1)

  * •

misunderstood (1)

  * •

university-should-select-monitoring-system (1)

  * •

consult-outside-sources (1)

_unhelpful (1)_

  * •

lecture-format (1)

  * •

resubmissions-allowed (1)

  * •

have-to-review-recordings (1)

  * •

increased-bandwidth (1)

  * •

turn-it-in (1)

  * •

convenient (1)

  * •

pandemic-safety (1)

  * •

prefer-live-monitoring (1)

  * •

not-enough-staff-for-watching (1)

  * •

allows-monitoring (1)

  * •

scientific (1)

  * •

distractions (1)

_roommates (1)_

  * •

timed-exam-remotely (1)

  * •

exam-over-days (1)

_caught-cheating (1)_

  * •

could-use-audio-files-in-test (1)

  * •

fully-online-class (1)

  * •

not-paid-enough-to-learn-systems (1)

  * •

lockdown-browser (1)

  * •

pop-quizzes (1)

  * •

students-select-time (1)

  * •

response-time (1)

_slow (1)_

  * •

controlled-testing-environment (1)

_during-virtual (1)_

  * •

faculty-mistrust-of-product (1)