# Characterizing active learning environments in physics using network
analysis and COPUS observations

Kelley Commeford  Eric Brewe  Drexel University
3141 Chestnut St, Philadelphia, PA 19104  Adrienne Traxler  Wright State
University
3640 Colonel Glenn Hwy, Dayton, OH 45435

###### Abstract

This study uses social network analysis and the Classroom Observation Protocol
for Undergraduate STEM (COPUS) to characterize six research-based introductory
physics curricula. Peer Instruction, Modeling Instruction, ISLE, SCALE-UP,
Context-Rich Problems, and Tutorials in Introductory Physics were
investigated. Students in each curriculum were given a survey at the beginning
and end of term, asking them to self-identify peers with whom they had
meaningful interactions in class. Every curriculum showed an increase in the
average number of student connections from the beginning of term to the end of
term, with the largest increase occurring in Modeling Instruction, SCALE-UP,
and Context-Rich Problems. Modeling Instruction was the only curriculum with a
drastic change in how tightly connected the student network was. Transitivity
increased for all curricula except Peer Instruction. We also spent one week
per research site in the middle of the term observing courses using COPUS.
From these observations, the student COPUS profiles look nearly the same for
Tutorials, ISLE recitations, and Context-Rich Problems discussion sections.
This is likely due to the large resolution of activities that can be coded as
“other group activity”, suggesting the need for a more detailed observation
instrument.

###### pacs:

Valid PACS appear here

††preprint: APS/123-QED

##  I Introduction

After decades of research in discipline based education research communities,
it is well established that active learning is more effective than standard
passive lecture at improving student outcomes Freeman  _et al._ (2014); Hake
(1998). A vast majority of active learning research still uses passive lecture
methods as a baseline measurement of learning techniques. As such, Freeman et.
al. recommends a second wave of discipline based education research
initiatives that study active learning methods independently of passive
lecture methods, so we can understand the mechanisms through which active
learning promotes increased student outcomes.

However, studies of active learning pedagogies as independent entities have
encountered difficulties because implementations vary widely. When comparing
learning gains of individual implementations of Tutorials in Introductory
Physics, with different instructors and student populations, there have been
inconsistent gains based on instructor background and student buy-in Pollock
and Finkelstein (2008); Finkelstein and Pollock (2005). Additionally, a study
of several Peer Instruction classrooms at the same university, but with
different instructors, shows that the implementation of the same pedagogical
methods vary greatly, resulting in differing classroom norms Turpen and
Finkelstein (2009). Implementation of a given active learning pedagogy can
also be severely limited based on external factors, such as assigned
classroom, class size, and student preparation Henderson (2008). Before we can
even think of directly comparing pedagogies to each other, we need to
understand the roles that different mechanisms play when implementing a
pedagogy, and what impact those mechanisms have on students interacting with
the curriculum.

To understand how active learning mechanisms impact student achievement, we
first need to develop an appropriate vocabulary to describe the distinguishing
features of an active learning environment. Doing so will allow researchers to
discuss these implementations as individual entities, rather than as umbrella
“active learning methods” to be compared to passive lecture.

Active learning environments vary based on student and instructor tasks and
behaviors. There have been a handful of observational characterization studies
done for Peer Instruction Wood  _et al._ (2016) and small group physics
workshops West  _et al._ (2013). This study aims to broaden that lens to
include six active learning pedagogies commonly used in physics, and we
propose to characterize these activities using two complimentary methods, the
Classroom Observation Protocol for Undergraduate STEM (COPUS) Smith  _et al._
(2013) and social network analysis.

Active learning, at its core, provides opportunities for students to interact
with each other. COPUS can be used to record instructor and student
interaction categories with the aim of creating a unique profile for each
implementation. Social network analysis can be used as a means to quantify
student interactions that arise during a given active learning implementation.
There is evidence that a student’s position in the classroom social network
improves course outcomes and persistence in physics Bruun and Brewe (2013);
Zwolak  _et al._ (2017), so it is natural to surmise that active learning
contributes to social network development.

With COPUS profiles and classroom social network data, we can begin describing
how classroom tasks and behaviors can correlate to student social network
development and mobility. By understanding what students are doing and how it
impacts their social position within the class, future research can aim to
better understand how different kinds of interactions lead to student growth.
The goal of this paper is to describe the characteristics of six distinct
pedagogies in physics, as measured by COPUS and network analysis. We do not
aim to directly compare pedagogies.

We begin with a brief literature review to introduce the observation protocol,
social network analysis concepts, and the active learning pedagogies that are
studied. Then, we describe how observation sites were chosen, the demographic
information of each site, and how data was collected. Finally, we show data
from each observed pedagogy and discuss how the in-class tasks and behaviors
present themselves in the social network data.

##  II Literature Review

In this section, we introduce the observation protocols that were considered
for this study, and discuss the chosen protocol (COPUS). We then introduce
social network analysis and define relevant terminology used throughout this
paper. Finally, we summarize the active learning pedagogies that are studied
in this project.

###  II.1 Observation Protocol

A research-based pedagogy is generally accepted as ‘active learning’ if the
students are involved in the learning process in some meaningful way.
Additionally, a pedagogy is considered ‘active learning’ if it is based on
research with regards to development and implementation, and students
consistently show learning gains when the pedagogy is implemented in the
classroom Meltzer and Thornton (2012). However, it is difficult to measure how
much active learning is occurring in a classroom, largely due to the wide
array of active learning methods that exist. The most straightforward way to
measure ‘active learning-ness’ is to conduct classroom observations using some
sort of protocol. Several observation protocols exist that serve various
purposes, some of which will be discussed briefly here.

There are two categories of observation protocols: open-ended and structured
Smith  _et al._ (2013). Open-ended protocols typically provide the observer
with prompting questions, to which they provide feedback. Structured protocols
provide the observer with some sort of framework to input observation data.
Since we aim to have concrete measurements of how much and what kind of active
learning is occurring, we discarded all protocols that were 100% open-ended,
as they rely on observer judgements and generally focus more on opinion
statements rather than measurable quantities. A discussion of structured
protocols that were considered for this study follows.

The Reformed Teaching Observation Protocol (RTOP) Sawada  _et al._ (2002) has
observers make holistic judgements about the quality of lesson design and
implementation, classroom culture, and content coverage. While it includes the
use of Likert scales to assign numerical values to these categories, these
categories are largely subjective. Additionally, the RTOP observation items do
not fully capture the time-dependence of student and instructor activities and
interactions. RTOP also has a long training time due to the in-depth
theoretical framework that it is built upon. Finally, RTOP gives a numeric
score where higher is better, which we wanted to avoid because our purpose is
comparison and description rather than ranking. The RTOP may be promising for
future studies delving into the quality of a given pedagogical implementation,
assuming inter-rater reliability could be reached.

The Real-Time Instructor Observation Tool, or RIOT, is a computer-based tool
that records instructor behaviors in real time West  _et al._ (2013). This
protocol gives a fine-grained temporal observation of instructor interactions
with students. While this tool is valuable for understanding how instructors
are leading conversations in the classroom, it was ultimately not chosen due
to the computational hardware requirement and lack of student observation
codes. This protocol is best suited for studying instructor behaviors
independently of student behaviors.

The Teaching Dimensions Observation Protocol (TDOP) Hora  _et al._ (2008) has
46 codes to delineate student and instructor behaviors in the classroom, as
well as a handful of open-ended responses. The observer codes interactions
during a two minute time interval; if the behavior occurred for longer than
five seconds, it gets coded. This protocol requires an extensive three-day
training to achieve inter-rater reliability. While this protocol gives a broad
overview of what is happening in the classroom, it was deemed impractical for
our needs due to long training and large number of codes to keep track of
during live observations.

The Classroom Observation Protocol for Undergraduate STEM (COPUS) Smith  _et
al._ (2013) is similar in structure to the TDOP, but has 25 codes instead of
46, and does not have observation codes that require the observer to make
judgements about the quality of instruction. It has been shown to have strong
inter-rater reliability after a mere one and a half hour training period. The
non-judgmental coding schemes provide a quantitative view of the classroom in
two-minute intervals. Due to the small number of codes, it is well known for
its ease of use in a live classroom observation Smith  _et al._ (2013).

We ultimately chose COPUS as the observation protocol for this study because
we wanted to get an overall picture of _what_ was happening in the classroom,
as opposed to how effective or well implemented the pedagogy was. In this
study, we intentionally chose development sites or well-regarded secondary
sites to ensure high-fidelity implementations, so the ‘instruction quality’
codes were deemed unnecessary for our initial investigation. Additionally, we
want our investigations to be easily reproducible, so the shorter training and
high inter-rater reliability were highly desirable. While COPUS is limited in
its ability to measure code duration, as codes are recorded in two-minute
intervals instead of instantaneously, it includes student and teacher behavior
codes that are appropriate for examining active learning pedagogies. Codes
included in the COPUS protocol can be seen in Table 1. Additional protocols
that address the time resolution concern were not appropriate or not published
at the time the project started.

Table 1: COPUS Codes for Instructor and Student Activities Smith  _et al._ (2013) Students are Doing |
---|---
L | Listening to instructor/taking notes, etc.
Ind | Individual thinking/problem solving. Only mark when an instructor explicitly asks students to think
| about a clicker question or another question/problem on their own
CG | Discuss clicker question in groups of 2 or more students
WG | Working in groups on worksheet activity
OG | Other assigned group activity, such as responding to instructor question
AnQ | Student answering a question posed by the instructor with rest of class listening
SQ | Students asks question
WC | Engaged in whole class discussion by offering explanations, opinions, judgment, etc. to whole class,
| often facilitated by instructor
Prd | Making a prediction about the outcome of demo or experiment
SP | Presentation by student(s)
TQ | Test or quiz
W | Waiting (instructor late, working on fixing AV problems, instructor otherwise occupied, etc.)
O | Other
Instructor is Doing |
Lec | Lecturing (presenting content, deriving mathematical results, presenting a problem solution, etc.)
RtW | Real-time writing on board, document projector, etc. (Often checked off along with Lec)
FUp | Follow-up/feedback on clicker question or activity to entire class
PQ | Posing non-clicker question to students (non-rhetorical)
CQ | Asking a clicker question (mark the entire time the instructor is using a clicker question, not just
| when first asked)
AnQ | Listening to and answering student questions with entire class listening
MG | Moving through class guiding ongoing student work during active learning task
1o1 | One-on-one extended discussion with one or more individuals, not paying attention to the rest of
| class (can be along with MG or AnQ)
D/V | Showing or conducting a demo, experiment, simulation, video, or animation
Adm | Administration (assign homework, return tests, etc.)
W | Waiting when there is an opportunity for an instructor to be interacting with or observing/listening
| to student or group activities and the instructor is not doing so
O | Other

###  II.2 Network Analysis

Social network analysis has been used as an analysis tool in a Modeling
Instruction classroom previously. Brewe et. al. showed that Modeling
Instruction produced classroom networks that were structurally unique from a
network formed in a standard passive lecture environment Brewe  _et al._
(2010), which was a large motivation for this project. Meanwhile, Zwolak and
Dou showed how social network analysis can be used to correlate student social
positioning with other factors, such as persistenceZwolak  _et al._ (2017) and
self-efficacyDou  _et al._ (2016). The survey methods deployed by Brewe et.
al. were refined by Zwolak and Dou, which subsequently influenced the survey
methods of this study.

Social network analysis has also been done in the context of upper division
physics courses, where homework grades were found to be strongly correlated
with student centrality in their homework problem-solving network. These
centrality measures remained stable across different courses and types of
assignments Vargas  _et al._ (2018). Vargas et. al. showed the potential
benefits of studying physics classrooms from a network perspective to
understand how student gains manifest.

We ultimately chose network analysis as a tool to study active learning
environments due to its applicability to relational data. Vargas and Brewe
demonstrated how social network analysis can be used in a physics context,
which we expand upon by collecting social network data from six pedagogies to
identify the network structures that arise when a given pedagogy is used.
Before we begin, we need to take a moment to define some network terms.

We have provided two toy networks to help illustrate these terms in figures 1
and 2.

Figure 1: A directed toy network, for illustration purposes only. The
direction of the arrows indicates the direction of interaction. Figure 2: An
undirected toy network, for illustration purposes only. If an interaction
occurred, the edge exists regardless of which node initiated the interaction.

Directed/Undirected – A network can be directed, meaning the edges have arrows
attached to them to illustrate which node initiated contact (such as in figure
1), or undirected, where the edge exists regardless of which node initiated
contact (such as figure 2). We use undirected networks in this study. The
following terms are defined in the context of undirected networks.

Node – Sometimes referred to as an actor, the node is the noun in the network.
In our toy network, the nodes are represented by the orange dots.

Edge – Sometimes referred to as a tie, an edge is the verb in the network.
Edges are represented by the lines connecting the nodes in the toy network
diagrams.

Degree – Degree measures how many edges a given node is connected with. For
example, in figure 2, node 1 is connected to nodes 2 and 4, for a total of two
connections. Node 1 has a degree of 2.

Density – The density is the ratio of actual number of edges in a network to
possible number of edges. In our undirected toy network (figure 2), there are
4 edges, but 10 possible unique edges, giving us a density of 2/5.

Diameter – If we calculate the geodesic distance (shortest path) between any
two nodes, and then do this for every pair of nodes in the network, the
longest of those paths is the diameter of the network. In our toy network
(fig. 2), our longest-shortest path is from 3$\rightarrow$1 via the path
3$\rightarrow$2$\rightarrow$1, for a diameter of 2.

Transitivity – In an undirected network, this is the ratio of closed triangles
to triads. A closed triangle in an undirected network looks like the
arrangement of nodes 1$\rightarrow$2$\rightarrow$4, while a triad would be one
step smaller and missing the closure, like 1$\rightarrow$2$\rightarrow$3\.
High levels of transitivity are usually indicative of a collaborative
environment, so this measure will be of particular importance when considering
network formation in active learning environments.

Giant component – The giant component is the number of nodes that are all
connected in the largest cluster of nodes. In our toy model, our largest
cluster has 4 nodes in it, so the giant component would be 4.

Isolate – An isolate is a node that is not connected to any other nodes, such
as node 5 in figure 2.

###  II.3 Active Learning Pedagogies

This project investigated six active learning pedagogies commonly used in
introductory physics courses at the university level. All of these pedagogies
have been featured in the New Faculty Workshops run by the American
Association of Physics Teachers Henderson (2008) and have research articles
describing their development, implementation, and outcomes (and can thus be
considered active learning according to Meltzer, et al.Meltzer and Thornton
(2012)).

####  II.3.1 Tutorials in Introductory Physics

Tutorials in Introductory Physics have been iteratively researched and
developed at the University of Washington McDermott and Shaffer (2002). This
curriculum is typically implemented in a traditional lecture/lab/recitation
setup, with the bulk of the tutorial material presented in recitation (aka
tutorial section). Tutorials focus on building a strong conceptual
understanding of the material before introducing calculations. The tutorial
curriculum materials consist of pre-tests, group worksheets, homework
problems, and post-tests. These materials are scaffolded and prompt students
to confront and resolve common misconceptions. Each tutorial section has one
or two teaching assistants who are trained to guide students through the
misconception confrontation process Scherr and Elby (2007).

####  II.3.2 ISLE

The Investigative Science Learning Environment (ISLE) approach was developed
at Rutgers University Etkina and Heuvelen (2007). The ISLE approach is
intended to be implemented in all parts of a course, but in some cases, it is
possible to use the ISLE philosophy only in a lab. The ISLE approach helps
students learn physics while treating them as neophyte scientists. Students
are encouraged to use an iterative process in their learning, much like they
would be expected to do in a scientific career. This process typically begins
with observing a simple, carefully chosen “observational experiment”.
Students, working in small groups, then try to explain the experiment based on
their observations, and use their explanation to make predictions about the
outcomes of new “testing” experiments that they design. When there is a
mismatch between the prediction and the outcome of the testing experiment, the
students revise the explanation. Multiple explanations are encouraged for the
observational experiments. To develop and test explanations, students use
multiple representations. Unlike several of the other curricula studied in
this project, ISLE focuses on building up student’s correct intuition rather
than debunking misconceptions. It is used in large and small enrollment
college physics courses and in many high school physics courses Etkina (2015).

####  II.3.3 Modeling Instruction

Modeling Instruction for university physics was developed collaboratively Phy
, modeled after the high school Modeling Instruction curriculum developed by
Wells and Hestenes Wells  _et al._ (1995). Modeling Instruction is ideal for
large, open classrooms with the ability for large group collaboration. It is a
curriculum that focuses on having students build their fundamental
understanding of physics from the ground up, by observing phenomena and then
creating models to describe the phenomena. Students use multiple
representations to explain their models, and deploy those models to future
situations until they break. A typical day in class begins with a cooperative
group activity. After the activity, students then engage in ‘whiteboard
meetings’, where they circle up with several other groups to share their
results. This forces students to consolidate their ideas into an
understandable, presentable format, and allows discussion between groups Brewe
(2008). There is very little lecture instruction; all material is learned
through the activities and model building.

####  II.3.4 Peer Instruction

Peer Instruction was popularized by Eric Mazur at Harvard University Mazur
(1997). Peer Instruction is typically used in large lecture halls as a way to
integrate active learning into traditional lecture-style courses, but can also
be used in smaller courses. A typical cycle of instruction begins with
approximately ten minutes of lecture followed by a clicker question. The
question is posed, students answer individually, students discuss with their
neighbors, and are sometimes allowed to answer again. This curriculum is
commonly facilitated with personal-response systems, such as clickers, color-
coded cards, or an online response program. While Peer Instruction is meant to
refer to a specific routine and style of questioning Crouch  _et al._ (2007),
the implementation of Peer Instruction varies wildly between instructors
Turpen and Finkelstein (2009).

####  II.3.5 Context-Rich Problems

Context-Rich Problems refers specifically to the Minnesota Model for Large
Introductory Courses that was developed as a physics curriculum at the
University of Minnesota Uni . Context-Rich Problems works within a standard
course structure consisting of lectures, labs, and recitation/discussion
sections. This pedagogy uses the cognitive apprenticeship model Collins  _et
al._ (1991) with an emphasis on problem solving skills as a means to organize
content in the course. For example, during lectures, the instructor solves
problems using an expert-like framework, illuminating the hidden decision-
making processes that are necessary in physics problems. During the labs and
discussion sections, students practice solving problems while giving and
receiving coaching from instructors and other students. This practice takes
place in groups of 2 to 4, structured by the principles of cooperative group
work Johnson and Johnson (1989).

The context-rich problems that students encounter with this pedagogy differ
from typical textbook problems in that they provide a realistic reason for
calculating something. Students are encouraged to follow the same expert-like
problem solving strategy their instructor demonstrates in lecture Heller and
Hollabaugh (1992). These types of problems typically contain extraneous
information, require the use of estimation, or require students to recall
commonly known values. All of these activities immerse students in a “culture
of expert practice” similar to a traditional apprenticeship.

####  II.3.6 SCALE-UP

Student Centered Activities for Large Enrollment Undergraduate Programs, or
SCALE-UP , was developed by Robert Beichner et al Beichner  _et al._ (2007).
It is an integrated learning environment designed for large-enrollment physics
classes with up to 100 students. The goal of SCALE-UP was to take a studio-
style environment and make it accessible for larger courses. A typical SCALE-
UP course does not include much lecture time, but instead relegates
information transfer to assigned readings outside of class. This leaves class
time for cooperative group problem solving, experiments, or answering
questions that students may have. SCALE-UP refers more to the environment than
the specific pedagogy; instructors are able to implement any pedagogy they
wish, and have it easily translated to a large class via the room layout. A
typical SCALE-UP classroom has large, round tables, capable of holding up to
nine students. Within these tables, students are in teams of 3 with whom they
solve problems and work on experiments. Having multiple teams at the same
table allows for group-to-group interaction without wreaking havoc on
classroom management. The room can either have whiteboards along the perimeter
wall or individual whiteboards at the tables. These are used for group problem
solving and sharing with the class. SCALE-UP has one session with the
students, there is no separate lab or recitation section; everything is done
in the same room when the lesson calls for it.

##  III Methodology

In this section, we describe how research sites were chosen and provide an
overview of the selected site demographics. We broadly describe the social
network and COPUS data collection protocols, then further describe the methods
for each site. We also include descriptions of each pedagogical
implementation.

###  III.1 Research Site Selection

To determine which curricula to study, we began by reviewing past New Faculty
Workshops, hosted by the American Astronomical Society, the American Physical
Society, and the American Association of Physics Teachers. There were three
main criteria we looked for when deciding on pedagogies: Does the pedagogy
have developed curriculum materials? Does the pedagogy have an established
body of research? Is this pedagogy still widely used in actual physics
classrooms?

After identifying the six pedagogies used in this study, we reached out
directly to the developers of the curriculum to identify high-fidelity
implementations of each pedagogy. When possible, the institution that
developed the pedagogy was used, however, this was not always feasible.
Secondary institutions were identified via recommendation of instructors with
extensive training or research experience with the pedagogy in question.
Research sites were compensated for participating in this study. Demographic
information for the chosen sites can be seen in Table 2.

Table 2: Institution-level demographics of research sites as provided by Integrated Post-Secondary Education Data System (IPEDS) U.S. Department of Education. Institute of Education Sciences, National Center for Education Statistics (2018). Class-level demographics were not available for this study. For demographics that do not sum to 100%, the missing entries are ‘unknown’.  | Tutorials | ISLE | Peer Inst. | Context-Rich | Modeling | SCALE-UP
---|---|---|---|---|---|---
| 4-year | 4-year | 4-year | 2-year | 4-year | 4-year
| Public | Public | Private | Public | Public | Public
Undergraduate Enrollment | 32,099 | 36,039 | 15,724 | 20,000 | 48,818 | 11,425
Female | 53% | 50% | 48% | 55% | 57% | 46%
Male | 47% | 50% | 52% | 45% | 43% | 54%
Full time | 92% | 94% | 89% | 30% | 57% | 90%
Part time | 8% | 6% | 11% | 70% | 43% | 10%
American Indian or Alaskan Native | 0% | 0% | 0% | 1% | 0% | 1%
Asian | 25% | 27% | 18% | 5% | 2% | 1%
Black or African American | 3% | 7% | 7% | 2% | 12% | 3%
Hispanic or Latino | 8% | 13% | 7% | 36% | 67% | 3%
White | 38% | 38% | 52% | 46% | 8% | 86%
Two or more races | 7% | 4% | 4% | 5% | 2% | 3%
Race or ethnicity unknown | 3% | 2% | 3% | 5% | 1% | 1%
Non-resident alien | 15% | 9% | 11% | 1% | 6% | 2%
24 and under | 92% | 92% | 86% | 62% | 76% | 93%
25 and over | 8% | 8% | 14% | 38% | 24% | 7%
In state | 62% | 83% | 44% | 93% | 84% | 35%
Out of state | 21% | 7% | 46% | 0% | 3% | 64%
Foreign country | 16% | 9% | 9% | 4% | 12% | 1%

###  III.2 Social Network Data Collection

Research sites provided rosters of students enrolled in the introductory
physics course for the term that would be observed. Students were invited via
email to complete an online survey during the first week of the term and again
at the end of the term. Students were asked to select their name from a list
of people enrolled in the class. Given the same list, they were presented with
the question

> “Please choose from the list of people that are enrolled in your physics
> class the names of any other student with whom you had a meaningful
> interaction in class during the past week, even if you were not the main
> person speaking.”

Students self-identified what counted as a ‘meaningful’ interaction. This
approach to network data collection has been used successfully in physics
education research previously Zwolak  _et al._ (2017); Dou  _et al._ (2016);
Brewe  _et al._ (2010). Instructors were asked to promote the survey in class
to encourage participation.

Students were included in the social network if they filled out the survey, or
were named by someone who filled out the survey in either the pre or post
distribution. If a student did not fill out the survey, or was not named by
someone else in either the pre or post distribution, they were not included in
the data. Additionally, students under 18 were not included as respondents due
to IRB restrictions, but they could still be included in the network by
someone else naming them.

###  III.3 COPUS Data Collection

Each institution was visited for a full week during the term to conduct COPUS
observations for as many sections as possible. Every two minutes, instructor
and student codes were marked if said code occurred during that interval.
COPUS profiles were created by taking the fraction of time slots observed
versus the total length of the observation, and recorded as a percentage of
class time for each code.

###  III.4 Tutorials in Introductory Physics: Data Collection

The tutorials site was an R1 institution in the Northwestern United States.
The course format included a lecture section, a laboratory section, and a
tutorial section. The network surveys were distributed by lecture section, as
all students in a lecture section were distributed into the same subset of
tutorial sections. In the survey, names were grouped by tutorial section to
facilitate students selecting peers they worked with in their tutorial class.
Three lecture sections were included in this study, which correlated to 24
tutorial sections. The tutorial sections had approximately 20 students per
section. Nineteen tutorial sections and one lecture section were observed.

All tutorials occurred in the same classroom, and attendance was graded.
Students were seated at small tables designed for four students, but group
sizes ranged from two to five students. Students chose their own groups, and
did not necessarily have to work with the same people each week. The tables
had whiteboards in the middle to facilitate group discussion. Each section had
one lead teaching assistant and one grader. The grader spent the first few
minutes passing back graded papers, and then joined the lead TA as a teaching
assistant. For the purposes of the COPUS observations, the lead TA was
observed using instructor codes. Grader interactions were also coded, but not
included in analysis. Tutorial sessions were 50 minutes long, but observation
was only done for 20 minutes in the middle of the session.

###  III.5 ISLE: Data Collection

The ISLE site was an R1 public institution in the Mid-Atlantic region of the
United States. There are two variations of ISLE at this institution, both of
which were included in this study. The first is a lab-only implementation,
where ISLE is used in the lab course and accompanied by a standard lecture and
recitation. The second is a whole-course implementation, where ISLE principles
are used across lab, recitation, and lecture.

####  III.5.1 Lab-Only Implementation

In the lab-only ISLE implementation, the lab course is taken separately from a
lecture and recitation course. The lab does not have to be taken in the same
term as the lecture and recitation, so only the lab sections were observed
with COPUS and given network surveys. Each section had approximately 28
students enrolled. Twenty lab sections were surveyed from this implementation.
Of those, five sections were observed using COPUS.

The physics laboratory is a typical lab-bench set-up with one computer per
station. Students work in groups of 2-3 to complete the activity, and use
shared Google Docs to draft and submit their lab report. COPUS observations
were taken for the entire 3 hour session.

####  III.5.2 Whole-Course Implementation

The whole-class ISLE implementation consisted of a lecture component, and then
small lab and recitation sections where group activities were performed. All
sections of the whole-course implementation must be taken concurrently, so
network surveys were distributed for each of the nine recitation sections as
well as each of the nine lab sections. All students were in the same lecture,
so a lecture survey was not distributed. Both lab and recitation sections had
approximately 28 students per section. Two lecture sessions were observed with
COPUS, as well as four recitations and one lab.

The lecture was held in a stadium-seating auditorium style lecture hall. The
lab and recitations were held in a separate lab/recitation room, which was set
up with small round tables to encourage group cooperation. Each table held a
group of 4 students, where they worked in pairs and shared with the other pair
at the same table. On recitation days, the students worked through ISLE
workbook activities. Recitations were an hour and a half long. On lab days,
students performed experiments following the ISLE protocol. Labs were three
hours long. COPUS observations were taken for the full length of each session.

###  III.6 Modeling Instruction: Data Collection

The Modeling Instruction site was an R1 institution in the Southeastern part
of the United States. The course used a studio format (integrated lab and
lecture components) that met twice a week. Network surveys were distributed in
each individual class section. Three sections were included in this study,
with sizes ranging from 64 to 92 students. All three sections were observed
using COPUS.

All course activity occurred in the same room, which was a large open format
room with large tables for group activities. There was a large whiteboard and
projectors at the front of the room. Students were distributed among the large
tables in groups ranging from 2 to 6 people (assigned group size was 6 but due
to absences could be as small as 2 on any given day). Student groups were
assigned by the instructor and changed twice during the term. The tables had
whiteboards in the middle to facilitate group discussion; in addition, the
boards were used during ‘board meetings’.

Within the groups, students ranged from working collaboratively on the
assigned activity to working alone while sitting next to other students.
However, the groups had to create a whiteboard summarizing their work to
present to their peers during large whiteboard meetings, held 1-3 times during
the class. Even if students worked alone on the activity, they were required
to collaborate with their table-mates to create the whiteboard. During the
whiteboard meeting, 3-5 groups made a large circle to discuss their work. Each
group shared their whiteboards, and students discussed whether they obtained
the same answers/results. Each section had several teaching assistants; in the
COPUS observations, only the lead instructor was coded. Modeling Instruction
sections were two hours long, of which COPUS observations began 10-15 minutes
into the session to allow students to get settled.

Both days of instruction were aggregated into a single COPUS profile for each
section. We felt it produced a more meaningful “snapshot” of the curriculum to
provide a week-long observation rather than a single class period.

###  III.7 Peer Instruction: Data Collection

The Peer Instruction site was an R1 private institution in the Mid-Atlantic
region of the United States. The course format included a lecture, a
laboratory section, and a recitation section. The network surveys were
distributed by lecture section, since that is where the Peer Instruction
curriculum was implemented. One section of Peer Instruction was surveyed and
observed using COPUS, with an enrollment of 113 students.

Class was held in a large stadium seating lecture hall. Students sat wherever
they wanted. The laboratory and recitation sections were completely open
during registration, so it was possible to have students mixing between
lecture sections in the smaller lab and recitation sections. The survey
rosters only included students in the lecture section in which they are
enrolled in order to constrain the network. The network survey was distributed
to the students in class as part of a ”Learning Catalytics” activity, which
resulted in a significantly higher response rate. Lectures were 50 minutes
long, of which twenty minutes were observed using COPUS, in the middle of the
session.

###  III.8 Context-Rich Problems: Data Collection

The Context-Rich Problems site was a 2-year community college in the Mid-
Pacific region of the United States. The course format included a lecture,
discussion section, and lab section. There were two distinct sections of
students. Each section attended the same lecture and discussion, and then was
split into two lab courses. Network surveys were distributed at the
lecture/discussion level, with enrollments of 48 and 45 for each section.
COPUS observations were performed for both discussion sections, all four lab
sections, and two sessions of lecture per section, for a total of four lecture
observations.

The lab component was held in a traditional style lab room with lab benches
that had computers at each station. Each group had 3-4 students. At the
beginning of class, a question was projected on screen to help prime students
for the activity. They were instructed to use whiteboards to design their
experiment based on what they were trying to learn, and relate the data they
were going to take to the concepts learned in class. Lab sections were 170
minutes long, of which the entire time was coded with COPUS.

The lecture was held in a small lecture hall with slightly tiered rows of
stationary desks. The lecture was 80 minutes long, of which the entire time
was coded with COPUS. The discussion section was held in the same room as the
lecture. Despite the desks being immobile, students would physically turn in
their seats to work in groups of 2-3 on a worksheet. They were provided
whiteboards to facilitate discussion. An additional teaching assistant was
present– although not included in the COPUS observations; they spent the
entire time guiding discussions (MG and 1o1). Discussion sections were 50
minutes long, of which the entire time was coded. The students worked in the
same groups in the lab and discussion sessions. These groups were assigned by
the instructor and maintained for 3-4 weeks at a time before being reorganized
so that students could work with different peers.

###  III.9 SCALE-UP: Data Collection

The SCALE-UP site was a public R2 institution in the Great Plains region. The
course used a studio format that met three times a week. There was only one
section of the course, taught by one instructor, with an enrollment of 71
students.

The observed SCALE-UP curriculum was performed in a large room designed for
active learning. There were several large, round tables with microphones at
the center. The perimeter of the room was covered with white boards. There
were television screens around the room in place of a projector, to allow for
viewing of presentation materials from multiple angles. The instructor had a
wireless microphone to allow for mobility without sacrificing sound quality.

Class was an hour and fifteen minutes long, of which the entire time was coded
with COPUS. Three days of COPUS observations were aggregated into a single
COPUS profile, as we felt it better represented the curriculum to give a week-
long “snapshot” instead of a single class period.

##  IV Results

While five of the six curricula had data collected from more than one section,
we chose the section with the highest survey response rate for network
analysis. Table 3 depicts the calculated network metrics for the social
networks presented in this paper. These values will be discussed in detail in
section V.1.

Table 3: Network metrics for six active learning curricula in physics. The section with the highest response rate within each pedagogy is reported. Curriculum |  Pre/Post |  Diameter |  Avg. Degree |  Density |  Transitivity |  Giant Component |  Number of Nodes
---|---|---|---|---|---|---|---
Tutorials |  Pre |  13 |  1.370 |  0.00941 |  0.1340 |  41 |  147
Tutorials |  Post |  13 |  1.580 |  0.01080 |  0.2640 |  67 |  147
ISLE-Lab only |  Pre |  6 |  1.410 |  0.05410 |  0.2310 |  15 |  27
ISLE-Lab only |  Post |  7 |  1.930 |  0.07410 |  0.3160 |  13 |  27
ISLE-Whole Class Lab |  Pre |  3 |  1.750 |  0.07610 |  0.4140 |  5 |  24
ISLE-Whole Class Lab |  Post |  4 |  2.250 |  0.09780 |  0.5900 |  11 |  24
ISLE-Whole Class Rec |  Pre |  3 |  0.667 |  0.03330 |  0.0000 |  4 |  21
ISLE-Whole Class Rec |  Post |  4 |  1.430 |  0.07140 |  0.5560 |  9 |  21
Modeling Instruction |  Pre |  10 |  2.510 |  0.03390 |  0.1790 |  62 |  75
Modeling Instruction |  Post |  5 |  5.600 |  0.07570 |  0.2100 |  72 |  75
Peer Instruction |  Pre |  13 |  1.830 |  0.01740 |  0.2330 |  61 |  106
Peer Instruction |  Post |  12 |  2.230 |  0.02120 |  0.2310 |  80 |  106
Context-Rich Problems |  Pre |  8 |  1.910 |  0.04340 |  0.1980 |  28 |  45
Context-Rich Problems |  Post |  6 |  3.640 |  0.08280 |  0.2450 |  41 |  45
SCALE-UP |  Pre |  7 |  1.710 |  0.02510 |  0.3360 |  27 |  69
SCALE-UP |  Post |  8 |  4.090 |  0.06010 |  0.5170 |  58 |  69

###  IV.1 Tutorials in Introductory Physics

####  IV.1.1 COPUS observations

The tutorial section presented here was chosen at random from the subset of
tutorial sections included in the top level lecture section. In the tutorial
section, students worked in groups of 2-5 students. Within the groups,
students ranged from working as a fully collaborative group on the tutorial
worksheet assignment, to working alone while sitting next to other students.
Figure 3(a) shows the student codes to illustrate the overwhelmingly common
codes, SQ (student asks a question) and WG (working in groups). In these
observations, the SQ COPUS code was only marked when a student explicitly
raised their hand to ask the TA or grader a question, not questions during an
already in progress one-on-one session.

Figure 3(b) shows the instructor codes for this tutorial section to illustrate
the three overwhelmingly common codes. The TA moved around the room and
prompted students to work on the activity (MG) and stopped to have extended
discussions with the student groups (1o1). The observations for the tutorial
sections looked extremely similar, with small variations depending if the TA
walked around the room and interjected themselves into student groups in an
effort to drive conversation (coded as MG, moving around the room and guiding
discussions, typically followed by 1o1 when a longer discussion arose), or if
the TA waited at the front of the room until a student raised their hand with
a question (coded as W-waiting).

(a) Student COPUS codes for a tutorial section of Tutorials in Introductory
Physics curriculum. Students worked in small groups on tutorial worksheet
activity, and occasionally asked questions to the TA.

(b) Instructor COPUS codes for the tutorial section of Tutorials in
Introductory Physics curriculum. Tutorial sections had a lead TA and a
grader/TA. The lead TA was coded with COPUS and shown here.

Figure 3: COPUS profiles of one tutorial section in the Tutorials in
Introductory Physics curriculum. Observations were twenty minutes long out of
the fifty minute section.

The tutorial section presented here was taken from the same lecture section as
presented in the networks. However, the observed lecture was _not_ the same as
presented in the network plots, but had the same instructor, so we assumed
high levels of similarity between the two sections for COPUS.

The observed lecture took place in a traditional, auditorium stadium-seating
lecture hall with a large projector and chalk boards. The lecture began with a
group activity about vectors, then used a response collection system to give
students a short quiz. Students used phones or computers to answer the
questions, and were allowed to work together as long as they gave their own
answer. After the quiz, an interactive lecture was given using PowerPoint
slides. The COPUS codes for the lecture section can be seen in figures 4(a)
and 4(b).

(a) Student COPUS codes for the lecture portion of the Tutorials in
Introductory Physics curriculum.

(b) Instructor COPUS codes for the lecture portion of the Tutorials in
Introductory Physics curriculum.

Figure 4: COPUS profiles of the lecture portion of the Tutorials in
Introductory Physics curriculum. Observations were twenty minutes long during
the fifty minute section.

####  IV.1.2 Network data

The network data for Tutorials in Introductory physics was collected at the
lecture section level. The lecture section with the highest response rate is
presented here.

(a) Beginning of term social network for Tutorials in Introductory Physics.

(b) End of term social network for Tutorials in Introductory Physics.

Figure 5: Pre and post-term social networks for Tutorial in Introductory
Physics. The network survey was distributed at the lecture level. Small
groupings of students are visible, likely due to tutorial groups. The color
shading on these plots indicates which students filled out the survey.

Figures 5(a) and 5(b) show the pre and post-network diagrams, respectively.
The networks are characterized by groupings of 2-5 students, indicative of
small group structure. There were also larger chain-like patterns, likely due
to the geometry of the lecture hall and the interactivity of the lecture-based
activities. Additionally, we saw linking between smaller clusters, indicating
cross-group information transfer.

(a) Beginning of term social network for Tutorials in Introductory Physics,
colored by tutorial section.

(b) End of term social network for Tutorials in Introductory Physics, colored
by tutorial section.

Figure 6: Pre and post-term social networks for Tutorial in Introductory
Physics, colored by tutorial section. We see clustering by color, indicating
that students form connections with peers in the same tutorial section.

(a) Beginning of term social network for Tutorials in Introductory Physics,
colored by lab section.

(b) End of term social network for Tutorials in Introductory Physics, colored
by lab section.

Figure 7: Pre and post-term social networks for Tutorial in Introductory
Physics, colored by lab section. While there is still some clustering by
color, the tutorial section seems to be the main driver for student
connections.

Figures 6(a) and 6(b) were colored by tutorial section for the pre and post
network, respectively. We saw in both cases clustering by tutorial section,
indicating that students largely worked with their tutorial group to learn
physics, even in the lecture portion of the class.

Figures 7(a) and 7(b) were colored by lab section. There was still some
clustering by section, but not as distinct as the tutorial sections. Tutorial
and lab section overlap ranged between 1 and 6 students, which could explain
clustering consistent in both sections.

###  IV.2 ISLE: Lab-only implementation

####  IV.2.1 COPUS observations

Figure 8(a) shows the student COPUS codes for one section of the ISLE lab-only
implementation. The entire class period was spent working on the group lab
activity (OG). Students would occasionally raise their hands to ask for help
from the teaching assistant (SQ). In some sections, like the one shown here,
the TA spent a few minutes during class going over concepts used in the lab
activities, in which case the students listened (L) and answered questions
posed to the entire class (AnQ).

Figure 8(b) shows the instructor COPUS codes from the observed TA behavior.
The TA spent most of their time moving around the room and guiding the student
activities (MG), and frequently stopped at groups to engage them in
conversation about the activity (1on1). In some sections, like this one shown,
the TA spent a few minutes going over concepts used in the lab activities,
which was coded as Lec and RtW. During this short lecture time, the TA also
posed questions to the entire class (PQ) and answered questions that the
students had (AnQ). They also spent some time handing back papers and
discussing grades with students (Adm), waiting for students to raise their
hands (W), or talking to other TAs who happened to stop by (O).

(a) Student COPUS codes for the laboratory section of the lab-only ISLE
curriculum.

(b) Instructor COPUS codes for a laboratory section of the lab-only ISLE
curriculum.

Figure 8: COPUS profiles of a lab section in the lab-only ISLE curriculum.
Observations were of the entire three hour period.

####  IV.2.2 Network data

Figure 9(a) shows the early-term network diagram for the lab-only
implementation. While there was a large cluster of students, the structure
seemed to indicate that three students in particular were responsible for the
majority of student interactions (the darkest shaded nodes). Meanwhile, in the
post-network of figure 9(b), the groupings of three students were much more
pronounced, suggesting that lab groups were the dominant driver of student
interactions.

(a) Beginning of term social network for one lab section of the lab-only ISLE
curriculum.

(b) End of term social network for one lab section of the lab-only ISLE
curriculum.

Figure 9: Pre and post-term social networks for the lab-only implementation of
ISLE. The network survey was distributed within individual lab sections. The
color shading on these plots indicates which students filled out the survey.

###  IV.3 ISLE: Whole-course implementation

####  IV.3.1 Recitations COPUS observations

Figure 10(a) shows the COPUS codes for the students during the recitation
section of the whole-course ISLE implementation. The majority of the time was
spent working on the active learning workbook activity (WG). Students
occasionally raised their hands to ask the instructor a question (SQ), and
spent some time listening to the instructor go over midterm instructions and
instructions for registering for the next term (L).

Figure 10(b) shows the instructor COPUS codes for the recitation section. The
instructor spent most of the time moving between groups and guiding the
activity (MG), and frequently stopped for extended one on one discussions
(1o1). Some time was spent describing how to register for the next term and
midterm logistics (Adm). The remaining time was spent waiting for a student to
raise their hand with a question (W) or talking to the TA (O).

(a) Student COPUS codes for the recitation section of the whole-course ISLE
curriculum.

(b) Instructor COPUS codes for the recitation section of the whole-course ISLE
curriculum.

Figure 10: COPUS profiles of one recitation section of the whole-course ISLE
curriculum. Observations were of the entire 75 minute section.

####  IV.3.2 Recitation Network Data

The recitation sections for the COPUS and the network plots presented here are
not from the same section. We chose to present the network with the highest
response rate, which did not have a matching COPUS observation. Since all
sections were taught by the same instructor and used the same activities, we
assumed them to be similar enough to be considered side by side.

The early term network shown in figure 11(a) shows a largely isolated student
population. During recitation, students worked on workbook activities, so it
could be explained as students working alone despite being at the same table
as other students. In the end of term network in figure 11(b), we can see some
more prominent grouping, but still a large number of isolates.

(a) Beginning of term social network for one recitation section of the whole-
course ISLE curriculum.

(b) End of term social network for one recitation section of the whole-course
ISLE curriculum.

Figure 11: Pre and post-term social networks for a recitation section of the
whole-course implementation of ISLE. The network survey was distributed in the
individual recitation sections. Small pairings of students are visible, likely
due to partner collaboration on recitation activities. The color shading on
these plots indicates which students filled out the survey.

####  IV.3.3 Lab COPUS Observations

Figure 12(a) shows the COPUS student codes for the whole-class implementation
during the lab section. The majority of the lab course was spent working on
the assigned activity (OG). Students occasionally raised their hands to ask
the instructor a direct question (SQ). The instructor spent some time going
over midterm instructions, with students listening (L). At the end of the day,
students cleaned up their stations and prepared to go home (O). Figure 12(b)
shows the instructor codes for the lab section of the whole-course ISLE
implementation. The instructor spent the majority of the time moving around
the room and guiding the activity (MG), and frequently stopped to have
extended one-on-one discussions (1o1). Some time was spent describing the
midterm (Adm), talking with the TA (O), briefly going over a common
misconception (Lec), or waiting at the side of the room for a group to raise
their hands (W).

(a) Student COPUS codes for the lab section of the whole-course ISLE
curriculum.

(b) Instructor COPUS codes for a lab section of the whole-course ISLE
curriculum.

Figure 12: COPUS profiles of a lab section in the whole-course ISLE
curriculum. Observations were of the entire three hour period

####  IV.3.4 Lab Network Data

The network plots for the lab sections show very distinct groupings,
indicating that students did not speak to anyone outside of their assigned
groups (figure 13(a)). As the term progressed, the distinct groups remained
the dominant structure, but introduced some cross-group interaction (Figure
13(b)).

(a) Beginning of term social network for one lab section of the whole-course
ISLE curriculum.

(b) End of term social network for one lab section of the whole-course ISLE
curriculum.

Figure 13: Pre and post-term social networks for the lab section of the whole-
course implementation of ISLE. The network survey was distributed within the
individual lab sections. Small groupings of students are visible, likely due
to collaborative lab groups.

####  IV.3.5 Lecture COPUS Observations

During the lecture, students listened to the instructor (L) and worked
together on a short problem solving activity (OG). The instructor lectured (L)
and wrote on the chalkboard (RtW), and posed an extended question to the
entire class (PQ). Students were not expected to respond with clickers, but
instead put their answer on a paper to be handed in. The rest of the time was
spent following up the extended question (Fup) and talking about the midterm
(Adm). Network data was not collected at the lecture level.

(a) Student COPUS codes for the lecture section of the whole-course ISLE
curriculum.

(b) Instructor COPUS codes for the lecture section of the whole-course ISLE
curriculum.

Figure 14: COPUS profiles of one lecture section in the whole-course ISLE
curriculum. Observations were of the entire hour long section.

###  IV.4 Modeling Instruction

####  IV.4.1 COPUS observations

We present the section with the highest response rate on the network survey.
Two COPUS observations were done for each section. The two corresponding COPUS
observations were aggregated into one graph to represent a week worth of class
time. This particular section had 11 groups of students, ranging in size from
2 to 6 students. There were three teaching assistants. The first day of class
was spent working on a worksheet in the small groups, and the second day
extended that activity to include an investigative lab experiment.

Figure 15(a) shows the student COPUS codes for the Modeling Instruction
course. While students spent most of the time working in groups on their
worksheets (WG), they also spent time having whole class discussions via
whiteboard meetings (WC). The experiment was coded as OG for other group
activity. SQ was only coded when a student explicitly raised their hand to
draw a TA to the group.

Figure 15(b) shows the instructor COPUS codes. Most of the time was spent
moving around the room, guiding the activities (MG) and frequently stopping
for extended one on one discussions (1o1). During the whiteboard meetings, the
instructor would frequently lead the discussion by posing questions (PQ) to
the students or answering questions the students still had after the
discussion.

(a) Student COPUS codes for one section of the Modeling Instruction
curriculum.

(b) Instructor COPUS codes for one section of the Modeling Instruction
curriculum. Only the instructor was coded, despite the presence of 3-4
teaching assistants.

Figure 15: COPUS profiles for one section of the Modeling Instruction
curriculum. Both days of instruction, each two hours long, were combined to
create profiles representing a weeks worth of instruction.

####  IV.4.2 Network Data

The early-term network plot in figure 16(a) shows high levels of inter-
connectivity. However, there were still two distinct “islands”, only connected
to the main cluster via a single person. The late-term network shown in figure
16(b) shows a much higher level of inter-connectivity; the density of this
network more than doubled throughout the ten week period.

(a) Beginning of term social network for one section of the Modeling
Instruction curriculum.

(b) End of term social network for one section of the Modeling Instruction
curriculum.

Figure 16: Pre and post-term social networks for Modeling Instruction. Network
surveys were distributed by section. The pre-network is tightly connected, but
still has two distinct islands at the top and right side of the network
diagram. The post network shows a highly integrated class structure. The color
shading on these plots indicates which students filled out the survey.

###  IV.5 Peer Instruction

####  IV.5.1 COPUS observations

The Peer Instruction COPUS data had the greatest range of significantly
present activities recorded. Figure 17(a) shows the COPUS student codes for
the observed Peer Instruction section. Students spent most of the class
listening and taking notes (L), or responding to clicker questions. Clicker
questions were evenly split between individual thinking (IND) and group
discussion (CG), indicating that the instructor followed the suggested Peer
Instruction model for clicker questions, which included time for both
individual and group discussion during the answer portion. There were also a
few instances of students asking questions (SQ) and students being called on
to answer questions (AnQ). The rest of the time was spent waiting for the
instructor to enable the clicker response system (W).

The instructor COPUS codes can be seen in figure 17(b). The instructor spread
their time across multiple activities. Less than half of the class time was
spent lecturing and writing on the board (Lec and RTW). More than half of the
time was spent posing individual questions (PQ) or clicker questions (CQ).
After a question was posed, the instructor spent time following up by going
over the answer (Fup) and answering questions the students still had (AnQ).
For this class period, a lot of time was spent waiting for the students to
answer the questions (W) or working through technical issues and activating
the student response system (O). Administrative tasks were also performed
(Adm) in the form of handing back midterm exams while the students were
thinking through clicker questions.

(a) Student COPUS codes for one section of the Peer Instruction curriculum.

(b) Instructor COPUS codes for one section of the Peer Instruction curriculum.

Figure 17: COPUS profiles of one section of the Peer Instruction curriculum.
Observations were twenty minutes of the fifty minute long section.

####  IV.5.2 Network Data

Figure 18(a) shows the student network at the beginning of the term. There
were string-like structures in this network, likely caused by the geographical
restriction that students were sitting in rows and thus limited to
interactions with students in their immediate vicinity. Branches could be
explained by speaking to fellow students in front of or behind the student.
There were a large percentage of isolated students, as students were not
required to sit near or interact with their peers. The post-term network in
figure 18(b) showed a very similar structure, but with slightly larger levels
of connectivity, indicating that students were more likely to engage in
discussion with their peers towards the end of the term.

(a) Beginning of term social network for one section of the Peer Instruction
curriculum.

(b) End of term social network for one section of the Peer Instruction
curriculum.

Figure 18: Pre and post-term social networks for Peer Instruction. The network
survey was distributed at the lecture level. Small pairings of students are
visible, likely due to think-pair-share types of clicker questions. We also
see a high number of isolates, likely due to the open-seating arrangement that
does not require students to interact with their peers. Additionally, we see
long chains of students, likely due to the layout of the lecture hall. The
color shading on these plots indicates which students filled out the survey.

###  IV.6 Context-Rich Problems

####  IV.6.1 COPUS observations

COPUS observations were taken for each part of the Context-Rich Problems
curriculum. The discussion section observations can be seen in Figure 19.
Students spent the majority of the time working in groups on the worksheet
activity (WG). They frequently raised their hand to ask the TA/instructor a
question (SQ). The instructor occasionally made announcements about the
activity to the whole class, which students listened to (L). Whiteboards were
available to facilitate group cooperation, which were gathered and put away at
the beginning and end of class (O). The instructor spent the majority of the
time walking around the room and guiding the activity (MG), frequently
stopping to have extended discussions with student groups (1o1). The
instructor also handed back papers (Adm) and went over points of difficulty
with the activity to the whole class (Lec and Fup).

(a) Student COPUS codes for the discussion section of the Context-Rich
Problems curriculum.

(b) Instructor COPUS codes for the discussion section of the Context-Rich
Problems curriculum.

Figure 19: COPUS profiles of one discussion section in the Context-Rich
Problems curriculum. Observations were of the entire fifty minute section

The lab section observations can be seen in Figure 20. Students spent the
first portion of the class listening to the instructions (L). Students then
solved a context-rich problem in their groups that was related to the lab
activity (OG), and used the problem to predict the outcome of their experiment
(Prd). The lab activity was then performed for the rest of the period (OG).
When students had questions, they raised their hands to attract the instructor
to their group (SQ). There were limited numbers of some apparatus, so groups
would have to wait for them to become available (W). At the end of the class,
the activity was cleaned up for the next class (O).

The instructor began the class by giving instructions for the activity and
reviewing key concepts that would be used (Lec, RtW). They then posed a
context-rich problem to the students that was related to the activity and
discussed the experiment as a class (PQ, Fup). Once the activity began, the
instructor moved around the room (MG) and frequently stopped for extended
small group discussions (1o1). Equipment sometimes needed troubleshooting (O),
and papers were passed back to the students (Adm).

(a) Student COPUS codes for the lab section of the Context-Rich Problems
curriculum.

(b) Instructor COPUS codes for the Lab section of the Context-Rich Problems
curriculum.

Figure 20: COPUS profiles of the lab section in the Contex-Rich Problems
curriculum. Observations were of the entire three hour period.

The lecture section observations can be seen in Figure 21. The students
listened to the instructor (L), answered directed questions (AnQ), and
answered clicker questions individually and as pairs (CG, IND). Students also
asked the instructor questions with the whole class listening (SQ). When demos
were presented, students were asked to predict the outcome (Prd) and solve a
related context-rich problem in pairs (OG).

The instructor would lecture and write on the board (Lec, RtW) to present new
content, and to follow up questions (Fup). Numerous clicker questions were
posed (CQ), as well as non-clicker questions (PQ). The instructor would use
demonstrations to illustrate non-clicker Context-Rich Problems (D/V), and
answered individual student questions (AnQ). Papers were also handed back
during group thinking time (Adm).

(a) Student COPUS codes for the lecture section of the Context-Rich Problems
curriculum.

(b) Instructor COPUS codes for the discussion section of the Context-Rich
Problems curriculum.

Figure 21: COPUS profiles of the lecture section in the Context-Rich Problems
curriculum. Observations were of the entire ninety minute section.

####  IV.6.2 Network Data

Figure 22(a) shows the early-term network for one lecture/discussion section
of the Context-Rich Problems curriculum. There were a couple of small
groupings, likely from lab or discussion, while the rest of the class was
lightly connected. Figure 22(b) shows the late-term network, where students
were much more heavily connected.

(a) Beginning of term social network for one section of the Context-Rich
Problems curriculum.

(b) End of term social network for one section of the Context-Rich Problems
curriculum.

Figure 22: Pre and post-term social networks for Context-Rich Problems. The
network survey was distributed at the lecture/discussion level. Small
groupings of students are visible in the pre network, likely due to discussion
or lab groups. The post network is significantly more connected. The color
shading on these plots indicates which students filled out the survey.

###  IV.7 SCALE-UP

####  IV.7.1 COPUS Observations

The week-long observation period allowed for three class sessions to be
documented. The COPUS data shown in figures 23(a) and 23(b) was aggregated
over all three observation periods to represent a week’s worth of class time.

Similar to other curricula already discussed, students solved problems in
groups using a whiteboard (OG), as shown in figure 23(a). However, this
whiteboard was on the perimeter of the room, so students physically got up and
walked to the whiteboards. Students also spent time listening to short
lectures (L), answering clicker questions both alone and in their groups (IND
and CG), and asking questions to the instructor (SQ). When the student groups
were called on during a whiteboard or clicker activity, it was marked as AnQ.

The instructor codes shown in figure 23(b) hit all of the categories available
with COPUS. There was some time spent giving short lectures via powerpoint
(Lec), posing individual (PQ) and clicker questions (CQ), and following up
those questions (Fup) with discussion and sometimes whiteboard explanations
(RTW). During problem solving or clicker question time, the instructor moved
around the room (MG) and engaged in discussions with the individual groups
(1o1). During part of the lecture period, a short PhET Perkins  _et al._
(2006) simulation was shown (D/V).

(a) Student COPUS codes for the SCALE-UP curriculum.

(b) Instructor COPUS codes for the SCALE-UP curriculum.

Figure 23: COPUS profiles for the SCALE-UP curriculum. All three days of
instruction, each 75 minutes long, were combined to create profiles
representing a weeks worth of instruction.

####  IV.7.2 Network Data

The structure of the room is evident in the social network graphs, seen in
figures 24(a) and 24(b). The clustering of the students is indicative of the
large table set-up of the classroom.

(a) Beginning of term social network for the SCALE-UP curriculum.

(b) End of term social network for the SCALE-UP curriculum.

Figure 24: Pre and post-term social networks for SCALE-UP. The network survey
was distributed at the section level. Larger groupings of students are
visible, likely due to the large table dynamic within the SCALE-UP room. The
color shading on these plots indicates which students filled out the survey.

##  V Discussion

The goal of this project was to develop a vocabulary to describe active
learning pedagogies as individual entities. In this section, we will discuss
how the network metrics varied with curricula and noticeable trends. We also
discuss the overall trends within the COPUS profiles.

###  V.1 Network Analysis

Table 3 shows the calculated network metrics for the presented curricula.

Diameter is a metric that is limited by class size, so the absolute value of
the diameter should not be compared across curricula. However, the change in
diameter from the beginning to the end of term for each network varied with
curriculum. Most curricula had a change in diameter of $\pm$ 1-2 students.
This suggests that the overall style of interaction between the students
remained largely unchanged from the early to late-term networks. However, we
saw a large decrease in diameter with Modeling Instruction, indicative of a
significantly more tight-knit class community at the end of the term. Modeling
promotes not only in-group cooperation, but it also brings more students
together to discuss concepts via whiteboard meetings. Thus, a large change in
diameter could be a distinguishing feature of a Modeling Instruction
classroom.

Average degree, on the other hand, is not as limited by class size. The
average degree increased throughout the term in all curricula. This makes
sense qualitatively; as students became more familiar with their peers, they
formed more connections, regardless of curriculum choice. We do notice,
however, that the largest gains in average degree occurred with Modeling
Instruction (+3.09), SCALE-UP (+2.38), and Context-Rich Problems (+1.73). The
remaining curricula had gains ranging from 0.21-0.763, averaging less than one
new connection per person throughout the term. This result suggests that
Modeling, SCALE-UP, and Context-Rich Problems are curricula that foster
building new connections while retaining old connections made previously in
the term.

With an increase in average degree, the density of connections increased in
all pedagogies as well. However, since density is also directly correlated to
class size, we cannot make conjectures about pedagogical influence on this
metric.

Transitivity, which can be loosely described as the base-level of
collaboration between three students, increased for all curricula except Peer
Instruction, which remained approximately constant. All curricula have a focus
on collaborative group interactions, with the exception of Peer Instruction,
which instead promotes partner interactions. The spatial limitations of the
Peer Instruction classroom may have also inhibited transitivity growth, as the
ability for students to engage with more peers than are present in their
immediate vicinity was hindered. As such, a stable level of transitivity
throughout the term may be indicative of a Peer Instruction classroom.

The giant component in all but ISLE:lab-only networks increased at the end of
the term, indicative of more connectivity throughout the entire class. As with
diameter, the absolute value of the giant component depends on class size, so
change is more meaningful when discussing this metric.

While we did notice promising features during this descriptive analysis, it is
unclear whether certain features arise due to the curriculum itself or the
classroom layout. If we compare the post-network diagrams from Modeling
Instruction (Fig. 16(b)) and SCALE-UP (Fig. 24(b)), we see completely
different structures despite similar classroom layout. Both Modeling
Instruction and SCALE-UP had students sitting at large tables and working
collaboratively. However, the Modeling Instruction social network developed
into a tightly knit learning community, while the SCALE-UP social network
retained distinct groupings based on the students’ physical locations in the
room. This may be attributed to the addition of the whole-class whiteboard
meetings in Modeling, whereas SCALE-UP retains individual group identities
during discussions.

While this project has showed promise for a method of describing active
learning curricula independent of lecture methods, there are some serious
limitations that need to be addressed. For one, the need for students to
respond to a survey to develop social networks introduces opportunity for data
loss via poor response rates. In a few cases, the instructor provided time in
class to fill out the survey. Response rates for sections that were allotted
class time to fill out the survey were significantly higher than those that
did not. While network metrics are typically robust to missing data Smith and
Moody (2013); Smith  _et al._ (2017) due to the reciprocity of ties between
students, poor response rate can render an incomplete picture.

We combated low response rates in a few ways. Reciprocity of ties allowed us
to include members of the class who were underage or declined to participate
in the survey. This means that if a student did not fill out the survey, or
were later cut for being under 18, they could still be presented in the
network if someone else named them as a meaningful interaction. Second, we
used an undirected network, meaning that if person A named person B, the edge
between person A and person B existed, even if person B did not also name
person A. Finally, data cuts only included students that filled out either the
pre or post-survey, or were named in the pre or post-survey.

###  V.2 COPUS

At this level of descriptive analysis, we see that the student COPUS profiles
in Tutorials (Fig. 3(a)), ISLE recitation (Fig. 10(a)), and Context-Rich
Problems discussion (Fig. 19(a)) are very similar. COPUS only has two codes to
refer to student group activities; working in groups on a worksheet (WG) and
other group activity (OG). The code WG was appropriate for Tutorials, ISLE
recitation, and Context-Rich Problems discussion, rendering similar profiles.
However, all other group activities were indistinguishable within the OG
category. ‘Other group activity’ was coded for experiments, lab report
creation, white board collaboration, and non-clicker problem group discussion.
While two codes for small-group student collaboration may be appropriate for a
Peer Instruction setting, which COPUS was designed for, we lost the capability
to meaningfully distinguish curricula at the student-group level.

Similarly, the COPUS code ‘O- Other’ was marked for numerous student and
instructor activities. For the instructor, this code included instances of
bathroom breaks, conversations with the TA or another instructor, organizing
lab materials, eating a snack during an extended teaching block, and
troubleshooting equipment. For students, this code was used for clean-up,
gathering materials, miming unit-vectors, turning in homework, or taking short
breaks. The wide range of activities that can be classified as ‘O’ is
troublesome, as instructional activities can sometimes fall into this category
that is largely dominated by non-instruction behaviors.

##  VI Conclusion

Despite losing resolution at the student-group level, and the broad brush of
activities included as ‘other’, it is still possible that COPUS will be able
to differentiate between curricula as a whole; further study using latent
profile analysis is underway Stains  _et al._ (2018). However, while a
valuable tool for interactive lecture environments, COPUS failed to
distinguish student group activities. This could suggest that these pedagogies
are not as distinct as we like to think, or that COPUS is an inadequate tool
for making these measurements.

Network analysis illuminated possible distinguishing features, such as a large
decrease in network diameter with Modeling Instruction, and static
transitivity with Peer Instruction. Now that we see promise in using this
method to characterize active learning environments, a larger scale study
would be advised. Additionally, we plan to use exponential random graph
modeling to determine if these metric trends are coincidental or a feature of
the pedagogy Robins  _et al._ (2007).

Our goal with this project was to develop a vocabulary to discuss active
learning curricula independently of lecture, which we have begun by using
COPUS observations and network analysis. More in-depth analysis of both
network metrics and COPUS profiles will be featured in future papers.

##  VII Acknowledgements

We would like to thank the institutions that participated in data collection
and observation arrangements. We would also like to express our heart-felt
thanks to the instructors that allowed us to infiltrate their classrooms for a
week, with an additional thank you to those who also provided thorough and
thoughtful feedback on this paper. Lastly, we would like to thank the National
Science Foundation (DUE 1711017 and 1712341) for their generous support of
this project.

## References

  * Freeman  _et al._ (2014) S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith, N. Okoroafor, H. Jordt, and M. Pat Wenderoth, Proc. Natl. Acad. Sci. USA 111, 8410 (2014).
  * Hake (1998) R. R. Hake, Am. J. Phys. 66, 64 (1998).
  * Pollock and Finkelstein (2008) S. J. Pollock and N. D. Finkelstein, Phys. Rev. ST Phys. Educ. Res. 4, 010110 (2008).
  * Finkelstein and Pollock (2005) N. D. Finkelstein and S. J. Pollock, Phys. Rev. ST Phys. Educ. Res. 1, 010101 (2005).
  * Turpen and Finkelstein (2009) C. Turpen and N. D. Finkelstein, Phys. Rev. ST Phys. Educ. Res. 5, 020101 (2009).
  * Henderson (2008) C. Henderson, Am. J. Phys. 76, 179 (2008).
  * Wood  _et al._ (2016) A. K. Wood, R. K. Galloway, R. Donnelly, and J. Hardy, Phys. Rev. Phys. Educ. Res. 12, 010140 (2016).
  * West  _et al._ (2013) E. A. West, C. A. Paul, D. Webb, and W. H. Potter, Phys. Rev. ST Phys. Educ. Res. 9, 010109 (2013).
  * Smith  _et al._ (2013) M. K. Smith, F. H. M. Jones, S. L. Gilbert, and C. E. Wieman, CBE Life Sci. Educ. 12, 618 (2013).
  * Bruun and Brewe (2013) J. Bruun and E. Brewe, Phys. Rev. ST Phys. Educ. Res. 9, 020109 (2013).
  * Zwolak  _et al._ (2017) J. P. Zwolak, R. Dou, E. A. Williams, and E. Brewe, Phys. Rev. Phys. Educ. Res. 13, 010113 (2017).
  * Meltzer and Thornton (2012) D. E. Meltzer and R. K. Thornton, Am. J. Phys. 80, 478 (2012).
  * Sawada  _et al._ (2002) D. Sawada, M. D. Piburn, E. Judson, J. Turley, K. Falconer, R. Benford, and I. Bloom, School Sci. Math. 102, 245 (2002).
  * Hora  _et al._ (2008) M. T. Hora, A. Oleson, and J. J. Ferrare,  _Teaching Dimensions Observation Protocol (TDOP) User’s Manual_ (2008).
  * Brewe  _et al._ (2010) E. Brewe, L. H. Kramer, G. E. O’Brien, C. Singh, M. Sabella, and S. Rebello, AIP Conf. Proc 1289, 85 (2010).
  * Dou  _et al._ (2016) R. Dou, E. Brewe, J. P. Zwolak, G. Potvin, E. A. Williams, and L. H. Kramer, Phys. Rev. Phys. Educ. Res. 12, 020124 (2016).
  * Vargas  _et al._ (2018) D. L. Vargas, A. M. Bridgeman, D. R. Schmidt, P. B. Kohl, B. R. Wilcox, and L. D. Carr, Phys. Rev. Phys. Educ. Res. 14, 020112 (2018).
  * McDermott and Shaffer (2002) L. C. McDermott and P. S. Shaffer,  _Tutorials in Introductory Physics_ , 1st ed. (Prentice Hall, Upper Saddle River, NJ, 2002).
  * Scherr and Elby (2007) R. E. Scherr and A. Elby, AIP Conf. Proc. 883, 46 (2007).
  * Etkina and Heuvelen (2007) E. Etkina and A. V. Heuvelen, in  _Reviews in PER: Research-Based Reform of University Physics_ , Vol. 1, edited by E. F. Redish and P. J. Cooney (American Association of Physics Teachers, College Park, MD, 2007).
  * Etkina (2015) E. Etkina, Am. J. Phys. 83, 669 (2015).
  * (22) “PhysPort Methods and Materials: University Modeling Instruction,” .
  * Wells  _et al._ (1995) M. Wells, D. Hestenes, and G. Swackhamer, Am. J. Phys. 63, 606 (1995).
  * Brewe (2008) E. Brewe, Am. J. Phys. 76, 1155 (2008).
  * Mazur (1997) E. Mazur,  _Peer Instruction: A User’s Manual_ , 1st ed. (Prentice Hall, Upper Saddle River, NJ, 1997).
  * Crouch  _et al._ (2007) C. Crouch, J. Watkins, A. Fagen, and E. Mazur, in  _Reviews in PER: Research-Based Reform of University Physics_ , Vol. 1, edited by E. F. Redish and P. J. Cooney (American Association of Physics Teachers, College Park, MD, 2007).
  * (27) “University of Minnesota Physics Education Research and Development: Minnesota Model for Large Introductory Courses,” .
  * Collins  _et al._ (1991) A. Collins, J. S. Brown, and A. Holum, Am. Educ. 15, 6 (1991).
  * Johnson and Johnson (1989) D. W. Johnson and R. Johnson,  _Cooperation and competition: Theory and research_ (Interaction Book Company, Edina, MN, 1989).
  * Heller and Hollabaugh (1992) P. Heller and M. Hollabaugh, Am. J. Phys. 60, 637 (1992).
  * Beichner  _et al._ (2007) R. J. Beichner, J. M. Saul, D. S. Abbott, J. J. Morse, D. Deardorff, R. J. Allain, S. W. Bonham, M. H. Dancy, and J. S. Risley, in  _Reviews in PER: Research-Based Reform of University Physics_, Vol. 1, edited by E. F. Redish and P. J. Cooney (American Association of Physics Teachers, College Park, MD, 2007).
  * U.S. Department of Education. Institute of Education Sciences, National Center for Education Statistics (2018) U.S. Department of Education. Institute of Education Sciences, National Center for Education Statistics, “Integrated Postsecondary Education Data System (IPEDS),”  (2018).
  * Perkins  _et al._ (2006) K. Perkins, W. Adams, M. Dubson, N. Finkelstein, S. Reid, C. Wieman, and R. LeMaster, Phys. Teach. 44, 18 (2006).
  * Smith and Moody (2013) J. A. Smith and J. Moody, Soc. Netw. 35, 652 (2013).
  * Smith  _et al._ (2017) J. A. Smith, J. Moody, and J. H. Morgan, Soc. Netw. 48, 78 (2017).
  * Stains  _et al._ (2018) M. Stains, J. Harshman, M. K. Barker, S. V. Chasteen, R. Cole, S. E. DeChenne-Peters, M. K. Eagan, J. M. Esson, J. K. Knight, F. A. Laski, M. Levis-Fitzgerald, C. J. Lee, S. M. Lo, L. M. McDonnell, T. A. McKay, N. Michelotti, A. Musgrove, M. S. Palmer, K. M. Plank, T. M. Rodela, E. R. Sanders, N. G. Schimpf, P. M. Schulte, M. K. Smith, M. Stetzer, B. Van Valkenburgh, E. Vinson, L. K. Weir, P. J. Wendel, L. B. Wheeler, and A. M. Young, Science 359, 1468 (2018).
  * Robins  _et al._ (2007) G. Robins, P. Pattison, Y. Kalish, and D. Lusher, Soc. Netw. 29, 173 (2007).