# Meta Transfer Learning for Early Success Prediction in MOOCs

Vinitra Swamy  EPFL<EMAIL_ADDRESS>,  Mirko Marras  University of
Cagliari<EMAIL_ADDRESS>and  Tanja Käser  EPFL<EMAIL_ADDRESS>
(2022)

###### Abstract.

Despite the increasing popularity of massive open online courses (MOOCs), many
suffer from high dropout and low success rates. Early prediction of student
success for targeted intervention is therefore essential to ensure no student
is left behind in a course. There exists a large body of research in success
prediction for MOOCs, focusing mainly on training models from scratch for
individual courses. This setting is impractical in early success prediction as
the performance of a student is only known at the end of the course. In this
paper, we aim to create early success prediction models that can be
transferred between MOOCs from different domains and topics. To do so, we
present three novel strategies for transfer: 1) pre-training a model on a
large set of diverse courses, 2) leveraging the pre-trained model by including
meta information about courses, and 3) fine-tuning the model on previous
course iterations. Our experiments on $26$ MOOCs with over $145{,}000$
combined enrollments and millions of interactions show that models combining
interaction data and course information have comparable or better performance
than models which have access to previous iterations of the course. With these
models, we aim to effectively enable educators to warm-start their predictions
for new and ongoing courses.

Transfer Learning, Meta Learning, Student Success Prediction.

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
Ninth ACM Conference on Learning @ Scale; June 1–3, 2022; New York City, NY,
USA.††booktitle: Proceedings of the Ninth ACM Conference on Learning @ Scale
(L@S ’22), June 1–3, 2022, New York City, NY, USA††price: 15.00††isbn:
978-1-4503-9158-0/22/06††doi: 10.1145/3491140.3528273††ccs: Computing
methodologies Neural networks††ccs: Computing methodologies Transfer
learning††ccs: Applied computing E-learning

##  1\. Introduction

Massive Open Online Courses (MOOCs) and digital learning environments are
becoming immensely popular [1, 2]. Enrollments have surged to over $220$
million MOOC learners, with several courses on Coursera and EdX reporting
increases by a factor of $10$ due to the COVID pandemic [3, 2]. In 2021 alone,
$40$ million new learners signed up for a MOOC, with providers hosting
$3{,}100$ new courses and over $500$ new programs [2]. However, MOOCs tend to
suffer from high dropout and low success rates [4, 5]. Adaptive guidance can
hence increase retention and student performance. Methods for early success
prediction build the basis for interventions [6].

Consequently, a large body of work has focused on student success prediction
in MOOCs based on students’ interaction data, using logistic regression [7],
support vector machines [8], random forests [9], or long short-term memory
(LSTM) networks [10]. Prior work has also suggested feature sets for solving
the success prediction task. For example, [11, 12] extracted count-based
features (e.g. number of online sessions, number of videos watched). Others
extracted features related to the course attendance rate or the ratio of
videos watched [13, 14] and fine-grained video behavior (either through
explicit features or through feeding raw video events into an LSTM [15]). [16]
has showed that regular working patterns are essential for success in MOOCs.
Notably, most of these studies have performed a-posteriori or post-hoc
analyses. Only few works have addressed early success prediction for ongoing
courses. For example, [17] performed a meta-analysis of features across early
prediction models on flipped courses and MOOCs. [14] showed that video
engagement in the first week and mid-point of the course can be indicative of
student pass-fail grades. [18] used recent temporal patterns in novice
programming tasks to early intervene.

The vast majority of performance prediction literature has focused on
inference on the course the model is trained on, i.e. on training the models
on one portion of the students and then predicting on the remaining students
of the course. In case of first time smaller courses, training on the same
course might lead to overfitting. For early success prediction, using the same
course for training and inference is not practical as the model does not have
the data required for supervised training (the pass-fail label will only be
known at the end of the course). Thus, for these settings, success predictions
need to rely on models trained on other courses.

Transfer learning is a well-known machine learning paradigm, which refers to
training a model on one setting, and using that model to warm-start
predictions in a different context [19]. In the context of education, only a
few works have attempted to create generalizable models that can be
transferred between different settings. [20] extract latent feature
representations that are transferable over multiple MOOC courses. [21] and
[22] warm-start their model on past data in two different settings: MOOC forum
post classification and active learning tasks. [23] transferred predictions
across 5 undergraduate courses at the same university. Most relevantly, [24]
showed that learning generalized patterns of student behavior from multiple
MOOCs can improve predictive performance.

In this paper, we aim to tackle the problem of transferability across MOOCs
from different domains and topics, focusing on models for early success
prediction. We present and analyze three novel strategies to creating
generalizable models for this task. First, we train a BiLSTM model taking
students’ interaction data as input on a large collection of diverse MOOCs.
Our second strategy is based on the assumption that students’ interaction
behavior is influenced by the context and hence courses with similar
characteristics will have similar student interaction patterns. We suggest two
architectures for including course information into a model, one being an
extension of our previous BiLSTM model and the second one combining
interaction data and course information using attention layers. We frame
success prediction as a meta learning problem (see e.g. [25]) through viewing
broad domains of MOOCs as separate tasks i.e. predicting student success in a
Mathematics course is often very different from predicting student success in
an English course. Our third strategy extends the aforementioned models by
applying fine-tuning on previous course iterations.

Figure 1. Our transfer approach based on behavior and course metadata
features, from logs collection to success prediction.

We extensively evaluate our models on a large data set including $26$ MOOCs
and $145{,}714$ students in total. With our analyses, we address the following
three research questions: 1) Can student behavior transfer across iterations
of the same course and across different courses? 2) Is a meta learning model
trained on a combination of behavior and course metadata information more
transferable? 3) Can fine-tuning a combined model on past iterations of an
unseen course lead to better transferable models? Our results demonstrate that
a model statically combining behavior and meta information over multiple
courses can significantly improve over one-course transfer baselines
($11.45\%$ on average across multiple early prediction levels and courses).
Our implementation and pre-trained models111https://github.com/epfl-
ml4ed/meta-transfer-learning achieve at least a comparable and often better
performance to training a customized model based on previous iterations of
their course.

##  2\. Methodology

The problem we address in this paper can be formulated as a time-series
classification task that relies on students’ interaction data as well as meta
information about the course to predict early on whether a student will pass
or fail a course. Specifically, we are interested in improving the accuracy of
success predictors for students belonging to a previously unseen iteration of
a course or to a completely new course. For clarity, we formalize the
addressed problem before presenting the proposed approach.

###  2.1. Problem Formulation

We consider a set of _students_ $S^{c}\subset\mathbb{S}$, who are enrolled in
a _course_ $c$ (referred also as course iteration), which is part of the
online educational offering $\mathbb{C}$. As courses can be run multiple times
over the year, each course $c$ refers to one run of a _course set_ $C$. We
denote a sequence of iterations for a course set
$C=\\{c_{1},\ldots,c_{M^{C}}\\}$, with $M^{C}$ being the total number of
iterations for the course set $C\in\mathbb{C}$. In other words, we consider a
course set $C$ to be a set composed of subsequent course iterations (e.g.,
multiple runs of a linear algebra course). Each course $c\in C$ has a
predefined _schedule_ consisting of $N_{c}=|\mathbb{O}^{c}|$ learning objects
from a catalog $\mathbb{O}^{c}$. In our study, we assume that the latter can
be either videos or quizzes, but the notation can be easily extend to other
types (e.g., forum posts, textbook readings). Enrolled students interact with
the learning objects in the schedule, generating a time-wise clickstream. We
denote the _interactions_ of a student $s\in\mathbb{S}^{c}$ in a course $c$ as
a time series $I_{s}^{c}=\\{i_{1},\ldots,i_{K}\\}$ (e.g., a sequence of video
plays and pauses, quiz submissions), with $K$ being the total number of
interactions of student $s$ in course $c$. We leave these definitions very
general, allowing the length of each time series to differ, since our models
can accommodate this. We assume that each interaction $i$ is represented by a
tuple $(t,a,o)$, including a _timestamp_ $t$, an _action_ $a$ (videos: load,
play, pause, stop, seek, speed; quiz: submit), and a _learning object_
$o\in\mathbb{O}^{c}$ (video, quiz). We finally denote the _binary success
labels_ (pass/fail) for students $S^{c}$ in a course $c$ as
$\mathbb{L}^{c}=\\{l_{s_{1}},\ldots,l_{s_{|S^{c}|}}\\}$.

###  2.2. Transfer Framework for Success Prediction

As shown in Fig. 1, our approach to solving the transfer problem in early
success prediction222Our approach and resulting models can be easily leveraged
for other downstream educational machine learning tasks, e.g., grade
prediction and dropout prediction. consists of three steps: 1) collecting and
preprocessing the log data, 2) computing behavior features for each student
(based on log data) and meta features per course, and 3) building three
different deep learning models based on different combinations of behavior and
meta features.

####  2.2.1. Data Collection and Preprocessing

A large portion of students drop out of a course during the first few weeks
(e.g., there are students who just enroll in the course for watching a few
videos) [26, 27]. Therefore, it is possible to predict course success for
these early-dropout students with a high accuracy by simply looking at their
assignment (or homework/quiz) submissions and grades in the first course
weeks. Developing complex machine learning models for predicting on these
early-dropout students is inefficient. Furthermore, training a machine
learning model on a highly imbalanced data set (i.e., with a large number of
students failing the course) might lead to a biased model. For that reason,
when not stated otherwise, we refer to $S^{c}$ as the students in a course
$c$, whose course pass-fail label cannot be easily predicted with a shallow
machine learning model, trained on the assignments grades in the first few
weeks of the course. While we filter out so called early-dropout students, we
will later show (Sec. 3.4) that our predictors have high accuracy on these
students as well. Our setting consequently leads to a faster and more
effective optimization process. To identify the early-dropout students $s\in
S^{c}$ of each course $c$, we fit a Logistic Regression model on the
assignment grades of the first course weeks. Specifically, our input data for
the model is a matrix $G$ of shape $|S^{c}|\times w$, where $w$ is the number
of course weeks and the value $G(i,j)$ is the grade student $s_{i}$ received
on the assignment in course week $j$. In our case, $w=2$. We considered the
average score of graded assignments for that week in case of multiple graded
assignments per week scored non-attempted assignments with a $0$. We filter
out the students whose predicted probability of course failure
$\hat{p}_{s}>0.99$. We found the optimal threshold via a grid search over
$\\{0.96,0.97,\ldots,0.999\\}$, maximizing the model’s balanced accuracy.

Given the low success rates for many MOOCs, targeted interventions have the
potential to improve learning outcomes [28, 29, 30, 6]. We are therefore
interested in early success prediction, providing the basis for such targeted
intervention (e.g., offering additional support to students at risk of failing
the course). We define an early prediction level $e$ and consider students’
interaction data only up to that point in time 333Our experiments will show
early predictions for $40\%$ and $60\%$ of the course duration.. If a course
had a duration of $10$ weeks and we aimed to predict student success after
$e=40\%$ of the course duration, we would only consider student interactions
happening in the first four course weeks. We denote the interactions of a
student $s\in\mathbb{S}^{c}$ in a course $c$ up to a specific point in time
$e$ with $I_{s}^{c,e}$.

####  2.2.2. Feature Extraction

For each course iteration $c\in\mathbb{C}$, we extract a set of behavior
features for each student $s$ based on the student’s interaction data
$I_{s}^{c,e}$ for an early prediction level $e$. We also extract a set of meta
information about that course iteration $c$.

Behavior Features. To obtain a comprehensive representation of student
behavior, we combine multiple types of features according to student
interactions with videos and quizzes. More granularly, we consider the four
behavior feature sets identified as the ones with the highest predictive power
for success prediction in the context of MOOCs [17]444Given the size and
variety of the course data considered in our study, we included all features
of the four features sets, instead of considering only the specific features
identified as important in at least one course by [17]. We then let our
classifiers detect the patterns and hence relevant features for the considered
courses.. Formally, given the interactions $I_{s}^{c,e}$ generated by students
$S^{c}$ until a course week $w\in\mathbb{N}$ in course $c$, we produce a
matrix $H\subset\mathbb{R}^{|S^{c}|\times w\times h}$ (i.e, each feature in
the feature set is computed per student per week), where $h\in\mathbb{N}$ is
the dimensionality of the feature set. As in Sec. 2.2.1, $w$ is chosen based
on the early prediction level $e$. We focus on the following behavioral
aspects555These 45 features are explicitly detailed in Appendix B.:

  * •

Regularity [16] ($H_{1}$, shape: $|S^{c}|\times w\times 3$). These features
quantify the regularity of a student’s study habits, for instance by measuring
whether the student usually studies during the same time of a day or has a
preference for specific days of the week.

  * •

Engagement [11] ($H_{2}$, shape: $|S^{c}|\times w\times 13$). These features
monitor the engagement of students throughout the course, considering
indicators such as the total number of student clicks on weekends and on
weekdays, and the total number of sessions.

  * •

Control [31] ($H_{3}$, shape: $|S^{c}|\times w\times 22$). This feature set
measures the fine-grained video consumption per student, including features
such as the proportion of videos watched, re-watched, or interrupted, and the
standard deviation of these ratios.

  * •

Participation [17] ($H_{4}$, shape: $|S^{c}|\times w\times 7$). These features
are related to attendance on videos and quizzes based on the schedule, such as
the number of scheduled videos watched for that week and the number of quizzes
passed on the first try.

Given a course $c\in\mathbb{C}$, we extract the above features for all
students $S^{c}\subset\mathbb{S}$ and concatenate them to obtain the final
combined behavior features for students in that course, defined as
$H^{c}\in\mathbb{R}^{|S^{c}|\times w\times 45}$, with $H^{c}=[H_{1}\cdot
H_{2}\cdot H_{3}\cdot H_{4}]$ (the $\cdot$ denotes a concatenation). Due to
the different scales, we perform a min-max normalization per feature in
$H^{c}$ (i.e., we scale the feature between 0 and 1 considering all students
and weeks for that feature).

Meta Features. Our application scenario includes a large variety of courses
with different characteristics (e.g., topic, structure, duration). We assume
that the observed interaction patterns of a course do not only depend on the
population (i.e. the individual students attending the course), but also on
the specific characteristics of the course. In prior work [32, 33], we see
that passing in meta information representing differences in context improves
predictive performance of transfer. We therefore code relevant information
about each course. Formally, given a course $c\in\mathbb{C}$, we produce a
fixed-length representation $F\subset\mathbb{R}^{f}$ for each meta feature,
where $f\in\mathbb{N}$ is the size of the meta feature. Specifically, we
consider:

  * •

Duration ($F_{1}$, shape: $f=1$) represents the number of course weeks as per
the schedule provided by the instructor.

  * •

Level ($F_{2}$, shape: $f=3$) is a one-hot encoded vector representing the
level of the course. Our course data includes three different levels:
Bachelor, Master, and Propedeutic.

  * •

Language ($F_{3}$, shape: $f=2$) is a one-hot encoded vector representing the
language in which the course is taught. Courses included in this study, were
taught either in French and English.

  * •

Title ($F_{4}$, shape: $f=30$) is a word embedding vector of the title the
instructor assigned to the course.

  * •

Short Description ($F_{5}$, shape: $f\in[30,60]$) is a word embedding vector
of the short description of the course (one sentence about the main topic).

  * •

Long Description ($F_{6}$, shape: $f\in[30,60]$) is a word embedding vector of
the long description of the course ($2{-}5$ more detailed sentences about the
course content).

Figure 2. Model architectures adopted in our study.

The features representing the Title, the Short Description, and the Long
Description were obtained by translating the corresponding text to English via
the DeepL translation service and then fine-tuning a pre-trained FastText
model [34] on that specific meta information666We experimented with the size
of the latent vector space and the encoding frameworks and found that FastText
outperformed other out-of-the-box encoders for this use case (SentenceBERT
[35] and Universal Sentence Encoder [36]).. FastText is a state-of-the-art
model for word representation (i.e., representing words in a text as latent
vectors that can be used for machine learning classification tasks).
Intuitively, we expect the Duration, Language, and Level meta features to
encode similarities in student population across courses and the Topic and
Description meta features to reflect similarity on course content. Given a
course $c\in\mathbb{C}$, we extract the above features and concatenate them to
obtain the final combined meta features for that course. Formally, the meta
features for course $c$ are denoted as $F^{c}\in\mathbb{R}^{124}$, with
$F^{c}=[F_{1}\cdot\ldots\cdot F_{6}]$. Again, due to the different scales of
these meta features, the values in $F^{c}$ are min-max normalized per feature.

####  2.2.3. Classification

In our transfer setting, given a set of courses $C_{trn}\subset\mathbb{C}$, we
are interested in creating an accurate success prediction model trained on
$C_{trn}$ for a specific early prediction level $e$, which can be transferred
to new (unseen) courses or new (unseen) iterations of a course. We are
interested in creating models that can predict accurately on courses not seen
during training, i.e., $C_{trs}\subset\mathbb{C}$, with $C_{trs}\cap
C_{trn}=\emptyset$ and $C_{trs}\cup C_{trn}=\mathbb{C}$. For a given early
prediction level $e$, we investigate the three model architectures shown in
Fig. 2.

Behavior Only (BO). Our first model predicts student success based on behavior
features (and thus student interaction data) only. The architecture of this
model is illustrated in Fig. 2. Formally, the model input is defined as
$H=\cup\;H^{c}\;\forall\;c\in C_{trn}$. Given that each course in $C_{trn}$
potentially has a different number of weeks, we consider the maximum course
duration $\textbf{w}=\max(\cup\;w^{c})\;\forall\;c\in C_{trn}$ and pad
behavior features for shorter courses with $-1$. The input $H$ therefore has a
shape of $|S^{c}|$ $\times$ w $\times$ $45$. There features are then fed into
a neural architecture composed by a _Masking_ layer ($mask=-1$), a simple yet
effective BiLSTM layer777We experimented with traditional machine learning
models (e.g., Support Vector Machines, Logistic Regression, Random Forest) and
deep-learning models (e.g., Dense Fully-Connected Networks, RNNs, LSTMs, CNNs,
and BiLSTMs), and found that the _BiLSTM_ layer performs best against the
other baselines for our use case. with a loopback of $3$, and a _Dense_ layer
(with Sigmoid activation) with a hidden size of $1$. The model outputs the
probability the student will fail the course.

Behavior + Time-wise Meta (BTM). Our second model is based on the assumption
that students’ interaction behavior is influenced by the specific
characteristics of the course. We expect that courses with similar
characteristics in terms of structure (e.g., topic, duration) and level (e.g.,
Bachelor) exhibit more similar behavior patterns than courses with completely
different characteristics. Therefore, including meta features into the model
might improve transfer (i.e. predictive performance of the model on unseen
iterations or courses). The resulting model is illustrated in Fig. 2. We use
the exact same architecture as for the BO model; we only change the input data
of the model to be a concatenation of both behavior and meta features for each
week, formally denoted as $H=\cup\;H^{c}\cdot F^{c}\;\forall\;c\in C_{trn}$
(with $\cdot$ representing a concatenation operator). The meta features fed
into the model are the same for every week since they are computed per course.
The final input $H$ has a shape of $|S^{c}|$ $\times$ w $\times$ $169$ ($45$
for the behavior features and $124$ for the meta features). By feeding in the
meta features for every week, we allow the BiLSTM to directly learn from the
dependencies between meta features and behavior features over time.

Behavior + Static Meta (BSM). Similar to our second model, our third model
also relies on the assumption that students’ interaction behavior varies
dependent on the course characteristics. However, this third model is based on
the hypothesis that not all course features are equally important, i.e. some
characteristics might have a much influence on student behavior than others.
We hence add attention to this model. The architecture consists of separate
branches for the behavior feature and meta features, followed by a head
architecture that elaborates the combined behavior and meta features (Fig. 2).
The behavior features branch has the same architecture as the BO model, except
for the removal of the final _Dense_ layer.

The meta feature branch receives the meta features $F^{c}$ of the course $c\in
C_{trn}$ (shape: $|S^{c}|\times|F^{c}|$) as input and is composed of two
layers. The first layer is a Bahdanau _Attention_ layer [37] which returns the
weights for all features in $F^{c}$, representing the importance of each of
these features for the prediction, i.e., a vector of size $|F^{c}|$). The
second layer is a _Concatenation_ layer that combines both raw meta features
and their importance weights returned by the _Attention_ layer (shape:
$2\times|F^{c}|$). This attentive layer is essential for internal and external
model interpretability, making it possible to identify the most important
(meta) features for the prediction.

Behavior and meta features follow different scales and variation patterns and
are differently associated to each input instance (e.g., meta features are the
same for each student belonging to a given course, while behavior features
change for each student). Therefore, the head architecture combining both the
behavior branch and the meta branch does not direct rely on a concatenation of
the outputs of these two branches. Rather, the output of each branch is
individually projected to the same latent space, obtaining a latent vector
(shape: $256$) for the output of each branch. To this end, we use a well-known
_Projection_ block, composed by a sequence of GELU, Dense, Dropout, and
Normalization layers, as proposed by [38]. The two latent vectors are then fed
into a _Concatenation_ layer (shape: $2\times 256$). A second Bahdanau
_Attention_ layer returns a vector of size $2\times 256=512$ representing the
weights for all latent behavior and meta features. This second attentive layer
weights the overall importance of behavior and of meta information for each
prediction. Then, another _Concatenation_ layer combines both the latent
features and their importance weights (shape: $2\times 512=1024$). Finally,
the concatenated output is passed to a cascade of _Dense_ layers, with the
last one having a Sigmoid activation and a hidden size of $1$. This final
layer returns the probability the student will fail the course.

##  3\. Experimental Evaluation

We conducted experiments to understand the extent to which student behavior
transfers well on iterations of the same course or of a different course
(RQ1), the extent to which course meta information can improve model
transferability, in addition to behavior features (RQ2), and whether fine-
tuning the latter meta models on previous iterations of a course unseen during
training leads to higher performance on the last iteration of the same course
(RQ3). The dataset, optimization protocol, and experiments are described
below.

Title | Identifier | Iterations1 | Topic2 | Level${}^{3^{8}}$ | Language4 | No. Weeks5 | No. Students3 |  Passing Rate6 [%] | No. Quizzes7 |
---|---|---|---|---|---|---|---|---|---|---
Trn | Trs | Trn | Trs | Trn | Trs |
Comprendre les Microcontrôleurs | Micro | 4 | 0 | Eng | BSc | French | 10 | 3,974 | - | 26.9 | - | 18 |
Analyse Numérique | AnNum | 3 | 0 | Math | BSc | French | 9 | 1,471 | - | 51.5 | - | 36 |
Household Water Treatment and Storage | HWTS | 2 | 0 | NS | BSc | French | 5 | 2,438 | - | 47.2 | - | 10 |
Programmation Orientée Objet | OOP | 1 | 0 | CS | Prop | French | 10 | 797 | - | 38.1 | - | 10 |
Programmation en C++ | InitProgC++ | 1 | 0 | CS | Prop | English | 8 | 728 | - | 63.3 | - | 13 |
Digital Signal Processing | DSP | 4 | 1 | CS | MSc | English | 10 | 11,483 | 4,012 | 22.6 | 23.1 | 38 |
Villes Africaines | Villes Africaines | 2 | 1 | SS |  BSc/Prop9 |  En/Fr9 | 12 | 7,888 | 5,643 | 6.3 | 9.9 | 18 |
L’Art des Structures I | Structures | 2 | 1 | Arch | BSc | French | 10 | 278 | 95 | 57.7 | 66.3 | 6 |
Functional Programming | ProgFun | 1 | 1 | CS | BSc | French | 7 | 11,151 | 7,880 | 50.72 | 81.33 | 3 |
Launching New Ventures | Venture | 0 | 1 | Bus | BSc | English | 7 | - | 6,673 | - | 1.4 | 13 |
Éléments de Géomatique | Geomatique | 0 | 1 | Math | BSc | French | 11 | - | 452 | - | 45.1 | 27 |
1 Set abbrev. Trn: training; Trs: transfer.
2 Topic abbrev. Eng: Engineering; Math: Mathematics; NS: Natural Science; CS:
Computer Science; SS: Social Science; Arch: Architecture; Bus: Economics and
Business.
3 The values are computed after removing early-dropout students.4Level is
chosen by majority label in Trs or Trn. 5Language is chosen by majority label
in Trs or Trn.
6 Passing Rate is averaged over the courses in Trs or Trn weighted by number
of students. 7No. Quizzes is averaged over the courses in Trs or Trn.
8 Level abbrev. Prop: Propedeutic / Other; BSc: Bachelor; MSc: Master. 9 For
Villes Africaines, the / operator represents characteristics of courses in Trn
/ Trs.

Table 1. Detailed information about the courses included in our dataset. | 40% Early Prediction Level | 60% Early Prediction Level
---|---|---
| 1-1 Same1 | 1-1 Diff2 | N-1 Same | N-1 Diff | 1-1 Same1 | 1-1 Diff2 | N-1 Same | N-1 Diff
DSP | 82.0 | 61.1 | 83.1 | 77.4 | 92.7 | 65.3 | 91.8 | 87.8
Villes Africaines | 73.8 | 64.6 | 69.4 | 79.7 | 82.9 | 67.0 | 80.7 | 82.7
Structures | 52.5 | 51.1 | 52.9 | 56.2 | 55.2 | 51.3 | 50.4 | 54.4
ProgFun | 50.6 | 50.7 | 59.5 | 58.7 | 50.8 | 51.0 | 62.3 | 62.0
Ventures | 51.0 | 59.8 | - | 70.3 | 54.9 | 60.2 | - | 71.8
Geomatique | 76.2 | 57.7 | - | 68.4 | 79.5 | 57.6 | - | 65.5
For each course and early prediction level, the highest value on a transfer
experiment is in bold.
1 1-1 Same is not included in determining the highest value across transfer
courses, since it could not exist in a real-world setting.
2 The balanced accuracy is averaged across the five trained models.
Table 2. Behavior transfer performance in terms of balanced accuracy for each
transfer course (the higher it is, the better).

###  3.1. Dataset

Our dataset includes $26$ MOOCs, taught by instructors of an European
university between 2013 to 2015. The data is fully anonymized with regards to
student information. In total, this dataset covers $145{,}714$ students. It
contains fine-grained video and quiz interactions for each student, e.g.,
pressing pause on a video or submitting a quiz. After removing the early-
dropout students (see Sec. 2.2.1), our data set contains $73{,}042$ students
in total. The variety of courses in the dataset (e.g., in terms of topic,
duration, level, and language), the large scale, and the varied population
(e.g., students come from different countries and cultures) allows us to
provide a realistic estimation of model performance. The courses are spread
across two languages, seven topics, and three university levels. The smallest
course has $95$ students, whereas the largest course has $11{,}151$ students.
Table 1 reports detailed information on each course as well as the assignment
to train and transfer sets. We will use these assignments for RQ1 and RQ2,
i.e. we will use $20$ courses as the train courses in $C_{trn}$ ($48{,}287$
students in total), and $6$ courses as the transfer courses in $C_{trs}$
($24{,}755$ students in total). To answer RQ3, we utilize another train-
transfer split: we will hold out one course $C$ completely (i.e. all
iterations of $C$ will be in $C_{trs}$) and train on all other courses.
Courses are split in train and transfer sets to ensure diversity in topic,
size, level, language, and passing rate.

###  3.2. Optimization Protocol

We trained each model with features $H$ for the courses in $C_{trn}$ under two
early prediction levels ($e=40\%$ and $e=60\%$), minimizing the binary cross-
entropy loss according to the setting outlined for each research question in
the next subsections. Grid search details are included in Appendix A. We
monitored BAC due to the high class imbalance888We computed accuracy, F1
Score, AUC, precision, and recall in addition to BAC, but we found these
metrics less representative of the predictive performance. . Formally, we
trained a model for each hyperparameter combination on the students in
$S_{trn}=\cup\;S^{c}-\tilde{S}^{c}\;\forall\;c\in C_{trn}$ and then selected
the best combination by measuring BAC on students in $S_{val}$. We then used
the selected model to predict student success and compute BAC for each
transfer course $c\in C_{trs}$.

###  3.3. Behavior Transfer Evaluation (RQ1)

Our initial experiment is designed to investigate the extent to which student
behavior transfers well on different iterations of the same course or of a
different course. We considered four main settings:

  * •

1-1 Same. For each transfer course $c\in C_{trs}$, we optimized a BO model on
a subset of the students ($80\%$) enrolled in the same course $c$. We then
predicted on the remaining set of students ($20\%$) in the same course $c$,
split into a test set and a validation set of $10\%$ each. We report results
using a 10-Fold cross validation, stratified on the pass-fail binary outcome
label. This setting is merely meant to give a measure of how difficult it is
to predict on a course and is not applicable in the real world, given that we
could not know in advance whether the students in the current iteration will
pass or fail the course.

  * •

N-1 Same. For each transfer course $c\in C_{trs}$ (with $c\in
C\subset\mathbb{C}$), we trained a BO model on the previous iterations of
course $c$, i.e., $(C\;-\;\\{c\\})\;\cap\;C_{trn}$. We then predicted on
students of course $c$.

  * •

1-1 Diff. For each transfer course $c\in C_{trs}$ (with $c\in
C\subset\mathbb{C}$), we optimized one separate BO model on each course
$\tilde{c}\in C_{trs}$, with $c\neq\tilde{c}$. For each model, we then
predicted on course $c$.

  * •

N-1 Diff. For each transfer course $c\in C_{trs}$, we optimized a single BO
model on all the training courses included in $C_{trn}$. We then predicted on
students of course $c$.

The BAC achieved by our BO models on each combination of transfer course and
training setting are listed in Table 2. For the $40\%$ early prediction level
(left section of Table 2), we observe that transferring a model from one
course to another (1-1 Diff) does not work well for any course. Training on
previous course iterations (N-1 Same) works best for the _ProgFun_ and _DSP_
courses. We hypothesize that this observation is because the latter two
courses include a large number of students ($>15{,}000$) in their previous
iterations and therefore, training the BO directly on the respective previous
iterations leads to better predictive performance. Training a BO model on a
large set of courses (N-1 Diff) on the other hand works best for _Villes
Africaines_ , which is a course with a very low passing rate ($<8\%$). We
assume that in case of highly imbalanced courses, the N-1 Diff model leads to
less biased predictions. The N-1 Diff model is also the best model for
_Structures_. This course contains a small number of students only
(${\sim}100$ per iteration) and the passing rates for the different iterations
vary a lot, which might explain the lower BAC of the N-1 Same model in
comparison to the N-1 Diff model. We also observe that for _Venture_ , the N-1
Diff model by far outperforms a model trained on students from the course
iteration (1-1 Same). _Venture_ is a course with a very low passing rate
($1.4\%$), likely leading to a high bias (towards failing students) in case of
the 1-1 Same model. The N-1 Diff model also performs well for _Geomatique_ ,
but we observe a transfer gap here with respect to a model trained on students
of the same course iteration (1-1 Same).

For the $60\%$ early prediction level (right section of Table 2), we make
similar observations. The N-1 Diff represents the best setting for _Villes
Africaines_ and _Structures_. The N-1 Diff is again outperformed by the N-1
Same setting for courses with the highest number of students ($>15{,}000$) and
a reasonably high passing rate ($>20\%$), i.e., _ProgFun_ and _DSP_. Also at
this early prediction level, the BO model trained under the N-1 Diff setting
represents the best real world solution on four out of six transfer courses.

We finally focus on the extent to which the balanced accuracy varies across
courses under the best performing transfer settings. For courses with a very
high number of students and a reasonably high passing rate, the best models
trained under the N-1 Same setting show a high variance in performance ($40\%$
level: $83.1$ BAC for _DSP_ and $59.5$ BAC for _ProgFun_ ; $60\%$ level:
$91.8$ BAC for _DSP_ and $62.3$ BAC for _ProgFun_). This difference in
transfer performance could be justified by the fact that _DSP_ includes
quizzes in the schedule (in contrast with _ProgFun_ , which only includes
videos) and thus the behavior features are more discriminative for the course.
The hypothesis that the lower performance is due to differences in the course
design is also supported by the low BAC of the 1-1 Same model, indicating that
_ProgFun_ is a hard course to predict on. For courses with a low number of
students and those with a high number of students but a low passing rate
(_Villes Africaines_ , _Structures_ , _Ventures_ , and _Geomatique_), the best
models under N-1 Diff show a more stable transfer performance ($40\%$ level:
avg. BAC $68.7$, std. dev. BAC $8.4$; $60\%$ level: avg. BAC $68.6$, std. dev.
BAC $10.3$).

(a) 40% early prediction level

(b) 60% early prediction level

Figure 3. Comparison of the balanced accuracy achieved by our behavior only
and behavior + meta models.

In summary, for courses with previous iterations, a very high number of
students, and a reasonably high passing rate, our BO model only on students
from these previous iterations (N-1 Same) works the best. In the other cases,
our BO model trained on students from the large set of courses (N-1 Diff) is
the most accurate. On average, the behavior transfer BAC is $75\%$ (57%) for
courses with a reasonably high (low) number of quizzes, under both early
prediction levels. On courses never seen during training, the behavior
transfer BAC is around $70\%$ under both early prediction levels.

###  3.4. Behavior + Meta Transfer Evaluation (RQ2)

In a second experiment, we were interested in investigating the extent to
which course meta information can improve model transferability. We therefore
experimented with how to best utilize the meta information for our educational
setting, trying different model architectures that pass in meta features at
different locations of the model (see Section 2.2.3). We also identified which
types of meta information were most relevant to the model through an ablation
study. We considered four main settings for the courses, as follows:

  * •

BO N-1 Same. For each transfer course $c\in C_{trs}$ (with $c\in
C\subset\mathbb{C}$), we optimized a BO model on the previous iterations of
course $c$ included in the training set, i.e.,
$(C\;-\;\\{c\\})\;\cap\;C_{trn}$. We then predicted on the students of course
$c$.

  * •

BO N-1 Diff. For each transfer course $c\in C_{trs}$, we optimized a single BO
model on all the training courses included in $C_{trn}$. We then predicted on
the students of course $c$.

  * •

BSM N-1 Diff. For each transfer course $c\in C_{trs}$, we optimized a single
BSM model on all the training courses included in $C_{trn}$. We then predicted
on the students of course $c$.

  * •

BTM N-1 Diff. For each transfer course $c\in C_{trs}$, we optimized a single
BTM model on all the training courses included in $C_{trn}$. We then predicted
on the students of course $c$.

(a) 40% early prediction level

(b) 60% early prediction level

Figure 4. Average performance of BO, BSM, BTM across transfer courses with and
without previous iterations, both on the filtered (w/ early-dropout students)
and the entire student population.

Predictive Power Analysis. Figure 4 illustrates the average predictive
performance of our models across all transfer courses. We computed the average
separately for courses with previous iterations in $C_{trn}$ and for courses
that do not have previous iterations. We observe that adding meta information
increases the average BAC for courses with previous iterations, no matter how
that meta information is added: the BSM (N-1 Diff) and the BTM (N-1 Diff) both
outperform the (BO (N-1 Diff)) model for both early prediction levels. For the
courses without previous iterations we again observe the same pattern for the
$40\%$ and $60\%$ early prediction levels. However, for these courses, the BSM
(N-1 Diff) model performs better than the BTM (N-1 Diff) model. It therefore
seems that for courses with previous iterations in $C_{trn}$, adding the
maximal amount of meta features helps the model focus on the similar courses
(i.e. the previous iterations of the transfer course) in the training data. In
contrast, for models without previous iterations, it seems to be important to
focus on only the relevant meta information (hence the better performance of
the BSM (N-1 Diff) model. With respect to training on previous iterations, we
observe that all the models trained on a large data set that include a range
of courses (BO (N-1 Diff), BSM (N-1 Diff), BTM (N-1 Diff)) show the same or a
higher BAC than the models trained on previous iterations of a course only (BO
(N-1 Same)).

The shaded bars in Fig. 4 denote the predictive performance of our models for
the whole student population (including early-dropout students). We observe
that every one of our models achieve a higher BAC for all students in both
early prediction settings. This result is expected as the early-dropout
students tend show very low activity already at the beginning of the course,
making it easy to predict on them. We also observe that the patterns of the
average BAC are the same when predicting on the whole population versus on the
filtered population. For the remaining experiments in this paper, we will
therefore report the BAC of the different models only on the filtered
population (as described in Sec. 2.2.1).

Figure 3 illustrates the BAC achieved by our models separately for each
transfer course. For the $40\%$ early prediction level (Fig. 3a), it can be
observed that adding meta information is helpful only for some courses of the
courses with previous iterations. For _DSP_ , adding meta information leads to
a strong increase in BAC: both meta models (BSM N-1 Diff and BTM N-1 Diff)
outperform the model including behavior features only (BO N-1 Diff). For
_ProgFun_ and _Structures_ , including meta information does not significantly
increase the BAC (in the case of the BSM N-1 Diff and BTM N-1 Diff models). In
the case of courses without previous iterations, adding meta information
increases the BAC for one course (_Ventures_), but leads to a slightly
decreased predictive accuracy for the second course (_Geomatique_). We further
observe that the BSM N-1 Diff model tends to outperform the BTM N-1 Diff
model. It performs better in four out of six courses. Furthermore, the BSM N-1
Diff model performs better than or at least as well as the model including
behavior features only (BO N-1 Diff) for four out of the six courses. We
observe a drop in performance only for two courses. Compared to the other
transfer courses, _Villes Africaines_ has a high number of students and a
highly imbalanced passing rate. The second course, _Geomatique_ , seems to be
hard to transfer to, as the performance of the BSM N-1 Diff model is very
close to the performance of the BO N-1 Diff model. For courses with previous
iterations, combining meta information and behavior features (BSM N-1 Diff)
led to the same or a higher BAC as the model trained on previous iterations of
the transfer course (BO N-1 Same).

For the $60\%$ early prediction level (Fig. 3b), the benefit of including meta
information is more evident. Except for _ProgFun_ and _Geomatique_ , both meta
models (BSM N-1 Diff and BTM N-1 Diff) have a BAC at least as high as the
model trained on behavior features only (BO N-1 Diff. In contrast to the
$40\%$ early prediction level, the performance differences between the BSM N-1
Diff model and the BTM N-1 Diff model are not as pronounced; the BSM N-1 Diff
model outperforms the BTM N-1 Diff model for three courses, but performs
(slightly) worse than the BTM N-1 Diff model for two other courses. Both the
BSM N-1 Diff model and the BTM N-1 Diff model achieve or exceed the predictive
performance of a model trained on previous course iterations (BO N-1 Same) for
three out of four courses. For _ProgFun_ , the predictive performance of the
two models is very close to that of the BO N-1 Same model.

Subsequently, we focus on the amount the BAC varies across courses under the
best performing setting, namely BSM N-1 Diff. On average, the BAC is higher in
all model settings for the $60\%$ early prediction level than for the $40\%$
level. This is expected because we would assume that the predictive power
generally improves with more available data about the student behavior. We
observe that the BSM N-1 Diff model shows a lower BAC for _Structures_ and
_ProgFun_ in comparison to the other courses. We have already observed in Sec.
3.3 that it seems to be generally hard to predict on the selected iteration of
the _ProgFun_ course (compare also the predictive performance of the 1-1 Same
model in Table 2). Both _Structures_ and _ProgFun_ include a comparably low
number of quizzes (see Table 1), we therefore hypothesize that the presence of
quizzes has a positive impact on model transfer.

(a) $40\%$ early prediction level

(b) $60\%$ early prediction level

Figure 5. Ablation study on the meta features for the BTM model. Figure 6.
Importance of the different meta features on the predictive power of the BSM
model.

Meta Information Importance Analysis. To have a more detailed picture of the
impact of the different meta features (see Section 2.2.2), we performed an
ablation study. We trained a BTM N-1 Diff model on $80\%$ of the students
attending the courses in $C_{trn}$ with only one meta feature at a time. We
then computed the BAC of the trained model on the remaining $20\%$ of the
students, split into a test and validation set. When performing the train-test
split, we stratified by course and pass-fail label. Figures 5a and 5b
illustrate the resulting BAC for the $40\%$ and $60\%$ early prediction
levels. The rightmost bar represents the BAC of a BO N-1 Diff model (no meta
information included). Notably, all the meta features improve the performance
of the model over the baseline BO N-1 Diff on both early prediction levels.
For the $40\%$ early prediction level, the most important meta features are
represented by Title (with an embedding size of $60$), Level, and Duration. It
therefore seems that Title contains enough information to help the model
leverage patterns from courses about similar topics. Furthermore, the ablation
importance of Level indicates that the interaction behavior of students might
differ according to their school level (e.g. Bachelor vs. Master). We theorize
that Duration is important since longer courses include more weeks of student
behavior data at the $40\%$ early prediction level. In the $60\%$ early
prediction level, the most important meta feature is Duration: there is a
minimal difference in importance for the other meta features. Based on the
results of this ablation study, we include all meta features into our models.
We however perform a grid search over Short/Long Description and Title (see
Sec. 3.2) to avoid training on highly redundant information.

To have additional insight into which meta features our model considers
useful, we examine the two _Attention_ layers of our BSM model. By analyzing
the importance weights resulting from these two layers, we can assess the
impact of the different meta features in combination with the behavioral
features on success prediction. We investigate these importance weights for
the BSM N-1 Diff model trained on the courses in $C_{trn}$ with all the meta
features. The importance weights are illustrated in Fig. 6 for both layers and
both the $40\%$and the $60\%$ early prediction levels. Considering the first
_Attention_ layer for the $40\%$ early prediction level (two leftmost plots)
which weights the meta features separately, it can be observed that on
average, Level appears as the highest weighted meta features, followed by
Title, Long Description, and Duration. As also observed in the ablation study,
Language appears to have lower importance than the other meta features at this
level. For the $60\%$ early prediction level, the weights assigned to Long
Description and Language increase, whereas the weights of Duration and Level
decrease. We hypothesize that when a model has less data about the students in
the course ($40\%$ early prediction level), it relies on heuristics about
Duration and Level for estimating the course structure and pass-fail patterns
and making predictive decisions. When more information about the course is
available ($60\%$ level), meta features like Long Description and Language
become slightly more relevant as the similarity of the course to other courses
in terms of these meta features is more useful to the model. Considering the
second _Attention_ layer (two rightmost plots), weighting all meta features
against all behavior features, it can be observed that the importance of the
behavior and meta features is, on average, equal. However, under the $60\%$
early prediction level, the variance of the importance for both types of
features is lower, i.e. there are less polarized cases where either behavior
features only or meta features only are important for a specific student.

In summary, combining behavior with meta features in a student prediction
model leads to consistently higher transfer performance across courses,
regardless of the early prediction level. Even when previous iterations are
available, the predictive power of meta models is comparable to that of a
model trained on the previous iterations of that course. In particular, a
static combination (BSM) is more transferable than a time-wise combination
(BTM). Level, Title, and Duration are important for very early predictions
(before mid-course), while all the meta features have a similar importance,
with Duration still being slightly more important, for later predictions.

(a) $40\%$ early prediction level

(b) $60\%$ early prediction level

Figure 7. Comparative balanced accuracy of different fine-tuned meta models.

###  3.5. Transfer-by-Fine-Tuning Evaluation (RQ3)

In a third and final experiment, we were interested in investigating the
extent to which fine-tuning behavior and meta models on previous iterations of
a course improves transfer. Related work from the broader computer vision
community has demonstrated that fine-tuning can improve performance in
transfer learning models [39]. To form an analogous setup for this experiment,
we held out a complete course $C$ (i.e. all iterations of that course) from
the training data. In a second step, we fine-tuned the obtained model on a
subset of iterations of $C$. Finally, we used this fine-tuned model to predict
on the last iteration of $C$. We selected three courses with different
characteristics (e.g., Duration, Topic) for our fine-tuning experiment and
hence $C_{trs_{ft}}=\\{Micro,AnNum,HWTS\\}$.

  * •

BO N-1 Same. For each transfer course $c\in C_{trs_{ft}}$, we optimized a BO
model on the previous iterations of course $c$, i.e. $C-c_{M^{C}}$. We then
predicted on the last iteration $c_{M^{C}}$ of course $C$.

  * •

BO N-C Diff. For each transfer course $C\in C_{trs_{ft}}$, we optimized a BO
model on the training data set $(C_{trn}-C)\cup C_{trs}$. We then predicted on
the students of the last iteration $c_{M^{C}}$ of course $C$.

  * •

BSM N-C Diff. For each transfer course $C\in C_{trs_{ft}}$, we trained a BSM
model on the training data set $(C_{trn}-C)\cup C_{trs}$. We then predicted on
the students of the last iteration $c_{M^{C}}$ of course $C$.

  * •

BTM N-C Diff. For each transfer course $C\in C_{trs_{ft}}$, we optimized a BTM
model on the training data set $(C_{trn}-C)\cup C_{trs}$. We then predicted on
the students of the last iteration $c_{M^{C}}$ of course $C$.

  * •

BO N-C Diff FT. For each transfer course $C\in C_{trs_{ft}}$, we fine-tuned
the optimal BO N-C Diff model (optimal on the training data set
$(C_{trn}-C)\cup C_{trs}$) on a subset of iterations of that course
$C-c_{M^{C}}$. We predicted on the last iteration $c_{M^{C}}$ of course $C$.

  * •

BSM N-C Diff FT. For each transfer course $C\in C_{trs_{ft}}$, we fine-tuned
the optimal BSM N-C Diff model (optimal on the training data set
$(C_{trn}-C)\cup C_{trs}$) on a subset of iterations of that course
$C-c_{M^{C}}$. We predicted on the last iteration $c_{M^{C}}$ of course $C$.

  * •

BTM N-C Diff FT. For each transfer course $C\in C_{trs_{ft}}$, we fine-tuned
the optimal BTM N-C Diff model (optimal on the training data set
$(C_{trn}-C)\cup C_{trs}$) on a subset of iterations of that course
$C-c_{M^{C}}$. We predicted on the last iteration $c_{M^{C}}$ of course $C$.

Figure 7 illustrates the BAC of the different models for the three courses and
two early prediction levels. For the $40\%$ early prediction level, we observe
that fine-tuning is helpful for only one course. For _HWTS_ , fine-tuning on
previous iterations of the course improves the BAC of behavior only (BO N-C
Diff) and combined meta-behavior (BSM N-C Diff and (BTM N-C Diff) models
significantly. For the other two courses, fine-tuning consistently decreases
the predictive performance of all investigated model. For the $60\%$ early
prediction level, the exact same patterns can be observed.

We postulate that the predictive performance increase achieved via fine-tuning
depends on the similarity (of student behavior) between the different
iterations of the course. The two iterations of _HWTS_ exhibit exactly the
same course characteristics in terms of topic, language, duration, and level.
Furthermore, both iterations of _HWTS_ contain a similar amount of students
with a very similar (balanced) passing rate ($45.5\%$ vs. $45.9\%$).
Therefore, training on previous iterations works well for this course; as
demonstrated in Fig. 7, the BO 1-Same model outperforms the other models for
both early prediction levels. For the other two courses, we observe that a
model trained on previous iterations of the same course exhibits a suboptimal
performance. In particular, the _AnNum_ course shows dissimilarities in terms
of passing rates over the years, i.e. going from a very low passing rate in
the first iteration ($8.5\%$) to passing rates $>70\%$ for iterations $2$ and
$3$. Indeed, we observe that training a model on previous iterations does not
work well for _AnNum_ (see BAC of the BO 1-Same model in Fig. 7), independent
of the early prediction level. Finally, _Micro_ changed over subsequent
iterations: the first run of the course started out with a lower number of
students ($571$ students), but was almost balanced in terms of pass-fail
labels (pass rate: $49\%$). The subsequent iterations saw a steep increase in
student count ($>4{,}000$ students), but also a large drop in the pass rate
(${\sim}5\%$). Training a model on a highly imbalanced course can lead to a
bias (towards failing students), while fine-tuning on previous iterations of
this course deteriorates model performance.

##  4\. Conclusions and Future Work

In this paper, we investigated the transferability of models trained on
different combinations of diverse student behavior and course meta features
for early student success prediction. In contrast to prior work [40], our
study aimed to exploit course datasets at scale to cover student prediction on
the last iteration of a given course and on courses with no previous
iterations available. To do so, we introduced three novel strategies: 1) pre-
training a model on a large set of diverse MOOCs, 2) including meta features
using two different architectures, one combining behavior and meta features
statically, and another one combining both in a time-wise fashion, and 3)
fine-tuning the aforementioned models on previous iterations of courses. We
then explored the extent to which student behavior transfers well on different
iterations of the same course or of a different course, the extent to which
the inclusion of course meta information can improve model transferability,
and finally whether fine-tuning on previous iterations of a course unseen
during training leads to higher performance on the last iteration of the same
course. This is the first comprehensive evaluation of model transferability
for early success prediction. The key implications are as follows:

  1. (1)

Behavior transfer on this task depends on the number of previous course
iterations, the number of students in those iterations, and their passing
ratio. If a course has past iterations with a high number of students and a
reasonably high passing rate, behavior features from the past iterations of
that course works best. Otherwise, behavior features from a large and diverse
course set leads to higher transferability. Few courses have low BACs across
all model architectures (transfer or individually trained), which reflects low
underlying signal in the data.

  2. (2)

Combining behavior and meta features using attention layers results in the
best transfer performance across courses, regardless of the early prediction
level. The resulting models and the models trained on the previous iterations
of a course (if available) have comparable or lower predictive power,
confirming findings of prior research on meta transfer learning in different
domains [33, 41, 42]. Our pre-trained static meta model could hence be
seamlessly applied to predict on new unseen courses.

  3. (3)

Predictive performance seems to be influenced by the number of quizzes in a
course. For both behavior only and meta models, the more quizzes the transfer
course included, the higher the BAC was. This implies that instructors who
consider utilizing student success predictors should carefully design their
course to include a good mix of lecture material and assessments.

  4. (4)

While all the meta features have similar importance for later prediction, the
course Level, Title, and Duration meta features are important for very early
predictions. Therefore, instructors should precisely define this course meta
information. Educational data scientists should include such meta features in
their models to foster transferability.

  5. (5)

When previous course iterations are available and similar (in student
population and course structure) to the current transfer course, fine-tuning a
meta model on these iterations can better transfer. However, if there are
significant differences between iterations, fine-tuning the model will lead to
a decrease in BAC.

Our results indicate that behavior and meta features could be successfully
used to improve model transferability. The models trained in this paper can be
used to warm-start performance predictions, aiming to improve student
performance in ongoing courses through targeted individual and curriculum
interventions [6]. Nevertheless, our work has several limitations that can be
addressed with future research. While we tested a variety of MOOCs with
different topics and populations on two early prediction levels, it is notable
the MOOC dataset is from a single university in Europe and collected five
years ago, leading to slight model bias because of the underlying data.
Further work is needed to account for different course modalities, such as
flipped classrooms and blended courses, to see the extent to which
transferability is modality-agnostic. Additionally, our study relies on hand-
crafted behavior features extracted from the raw clickstreams. Latent behavior
feature models directly acting on clickstreams (e.g., autoencoders) can be
combined with our architectures, to provide an end-to-end model and avoid the
need of manually extracting behavior features.

Acknowledgements. We kindly thank SERI and Professor Martin Jaggi for
supporting this project.

## References

  * [1] Sergey Kruchinin.  An investigation into the attraction and completion rates of moocs.  Knowledge Management & E-Learning: An International Journal, 11(1):38–58, 2019.
  * [2] Dhawal Shah.  By the numbers: Moocs in 2021.  2021\.
  * [3] Chris Impey and Martin Formanek.  Moocs and 100 days of covid: Enrollment surges in massive open online astronomy classes during the coronavirus pandemic.  Social Sciences & Humanities Open, 4(1):100177, 2021.
  * [4] Hanan Aldowah, Hosam Al-Samarraie, Ahmed Ibrahim Alzahrani, and Nasser Alalwan.  Factors affecting student dropout in moocs: a cause and effect decision-making model.  Journal of Computing in Higher Education, 32(2):429–454, 2020.
  * [5] Katy Jordan.  Mooc completion rates.  2015\.
  * [6] Mar Perez-Sanagustin, Ronald Pérez-Álvarez, Jorge Maldonado-Mahauad, Esteban Villalobos, Isabel Hilliger, Josefina Hernández, Diego Sapunar, Pedro Manuel Moreno-Marcos, Pedro J Muñoz-Merino, Carlos Delgado Kloos, et al.  Can feedback based on predictive data improve learners’ passing rates in moocs? a preliminary analysis.  In Proceedings of the Eighth ACM Conference on Learning@ Scale, pages 339–342, 2021.
  * [7] Jacob Whitehill, Kiran Mohan, Daniel Seaton, Yigal Rosen, and Dustin Tingley.  Mooc dropout prediction: How to measure accuracy?  In Proceedings of the fourth (2017) acm conference on learning@ scale, pages 161–164, 2017.
  * [8] Sabina Tomkins, Arti Ramesh, and Lise Getoor.  Predicting post-test performance from online student behavior: A high school mooc case study.  International Educational Data Mining Society, 2016.
  * [9] Mack Sweeney, Huzefa Rangwala, Jaime Lester, and Aditya Johri.  Next-term student performance prediction: A recommender systems approach.  arXiv preprint arXiv:1604.01840, 2016.
  * [10] Lisa Wang, Angela Sy, Larry Liu, and Chris Piech.  Deep knowledge tracing on programming exercises.  In Proceedings of the fourth (2017) ACM conference on learning@ scale, pages 201–204, 2017.
  * [11] Fu Chen and Ying Cui.  Utilizing student time series behaviour in learning management systems for early prediction of course performance.  Journal of Learning Analytics, 7(2):1–17, 2020.
  * [12] David John Lemay and Tenzin Doleck.  Grade prediction of weekly assignments in moocs: mining video-viewing behavior.  Education and Information Technologies, 25(2):1333–1342, 2020.
  * [13] Huan He, Qinghua Zheng, Bo Dong, and Hongchao Yu.  Measuring student’s utilization of video resources and its effect on academic performance.  In Proceedings of the 18th IEEE International Conference on Advanced Learning Technologies, ICALT 2018, pages 196–198. IEEE Computer Society, 2018.
  * [14] Boniface Mbouzao, Michel C. Desmarais, and Ian Shrier.  Early prediction of success in MOOC from video interaction features.  In Proceedings of the 21st International Conference on Artificial Intelligence in Education, AIED 2020, volume 12164 of Lecture Notes in Computer Science, pages 191–196. Springer, 2020.
  * [15] Ahmed Ali Mubarak, Han Cao, and Salah A. M. Ahmed.  Predictive learning analytics using deep learning model in moocs’ courses videos.  Education and Information Technologies, 26(1):371–392, 2021.
  * [16] Mina Shirvani Boroujeni, Kshitij Sharma, Łukasz Kidziński, Lorenzo Lucignano, and Pierre Dillenbourg.  How to quantify student’s regularity?  In European conference on technology enhanced learning, pages 277–291. Springer, 2016.
  * [17] Mirko Marras, Julien Tuang Tu Vignoud, and Tanja Kaser.  Can feature predictive power generalize? benchmarking early predictors of student success across flipped and online courses.  In 14th International Conference on Educational Data Mining, pages 150–160, 2021.
  * [18] Ye Mao.  One minute is enough: Early prediction of student success and event-level difficulty during novice programming tasks.  In In: Proceedings of the 12th International Conference on Educational Data Mining (EDM 2019), 2019.
  * [19] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang.  A survey of transfer learning.  Journal of Big data, 3(1):1–40, 2016.
  * [20] Mucong Ding, Yanbang Wang, Erik Hemberg, and Una-May O’Reilly.  Transfer learning using representation learning in massive open online courses.  In Proceedings of the 9th international conference on learning analytics & knowledge, pages 145–154, 2019.
  * [21] Xiaocong Wei, Hongfei Lin, Liang Yang, and Yuhai Yu.  A convolution-lstm-based deep neural network for cross-domain mooc forum post classification.  Information, 8(3):92, 2017.
  * [22] Shamya Karumbaiah, Andrew Lan, Sachit Nagpal, Ryan S Baker, Anthony Botelho, and Neil Heffernan.  Using past data to warm start active machine learning: Does context matter?  In LAK21: 11th International Learning Analytics and Knowledge Conference, pages 151–160, 2021.
  * [23] Maria Tsiakmaki, Georgios Kostopoulos, Sotiris Kotsiantis, and Omiros Ragos.  Transfer learning from deep neural networks for predicting student performance.  Applied Sciences, 10(6):2145, 2020.
  * [24] Jacob Whitehill, Kiran Mohan, Daniel Seaton, Yigal Rosen, and Dustin Tingley.  Delving deeper into mooc student dropout prediction.  arXiv preprint arXiv:1702.06404, 2017.
  * [25] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey.  Meta-learning in neural networks: A survey.  arXiv preprint arXiv:2004.05439, 2020.
  * [26] Daniel Onah, Jane Sinclair, and R Boyatt.  Dropout rates of massive open online courses: Behavioural patterns.  07 2014.
  * [27] Joselyn Goopio and Catherine Cheung.  The mooc dropout phenomenon and retention strategies.  Journal of Teaching in Travel & Tourism, 21(2):177–197, 2021.
  * [28] Inma Borrella, Sergio Caballero-Caballero, and Eva Ponce-Cueto.  Taking action to reduce dropout in moocs: Tested interventions.  Computers & Education, page 104412, 2021.
  * [29] Wanli Xing and Dongping Du.  Dropout prediction in moocs: Using deep learning for personalized intervention.  Journal of Educational Computing Research, 57(3):547–570, 2019\.
  * [30] Jacob Whitehill, Joseph Williams, Glenn Lopez, Cody Coleman, and Justin Reich.  Beyond prediction: First steps toward automatic intervention in mooc student stopout.  Available at SSRN 2611750, 2015.
  * [31] Sébastien Lallé and Cristina Conati.  A data-driven student model to provide adaptive support during video watching across moocs.  In International Conference on Artificial Intelligence in Education, pages 282–295. Springer, 2020.
  * [32] Seong-Jin Park, Seungju Han, Ji-Won Baek, Insoo Kim, Juhwan Song, Hae Beom Lee, Jae-Joon Han, and Sung Ju Hwang.  Meta variance transfer: Learning to augment from the others.  In International Conference on Machine Learning, pages 7510–7520. PMLR, 2020.
  * [33] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele.  Meta-transfer learning for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 403–412, 2019.
  * [34] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.  Enriching word vectors with subword information.  Transactions of the Association for Computational Linguistics, 5:135–146, 2017.
  * [35] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks, 2019.
  * [36] Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil.  Universal sentence encoder, 2018.
  * [37] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [38] Multimodal entailment in keras.  2022\.
  * [39] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris.  Spottune: transfer learning through adaptive fine-tuning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4805–4814, 2019.
  * [40] Sebastien Boyer and Kalyan Veeramachaneni.  Transfer learning for predictive models in massive open online courses.  In International conference on artificial intelligence in education, pages 54–63. Springer, 2015.
  * [41] Aqsa Saeed Qureshi, Asifullah Khan, Aneela Zameer, and Anila Usman.  Wind power prediction using deep neural network based meta regression and transfer learning.  Applied Soft Computing, 58:742–755, 2017.
  * [42] Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, Peng Xu, and Pascale Fung.  Meta-transfer learning for code-switched speech recognition.  arXiv preprint arXiv:2004.14228, 2020.

## Appendix

Set | Feature | Description
---|---|---
Regularity | DelayLecture | The average delay in viewing video lectures after they are released to students.
RegPeakTimeDayHour | The extent to which students’ activities are centered around a particular hour of the day.
RegPeriodicityDayHour | The extent to which the hourly pattern of user’s activities repeats over days.
Engagement | NumberOfSessions | The number of unique online sessions the student has participated in.
RatioClicksWeekendDay | The ratio between the number of clicks in the weekend and the weekdays
AvgTimeSessions | The average of the student’s time per session.
TotalTimeSessions | The sum of the student’s time in sessions.
StdTimeSessions | The standard deviation of student’s time in sessions.
StdTimeBetweenSessions | The standard deviation of the time between sessions of each user.
TotalClicks | The number of clicks that a student has made overall.
TotalClicksProblem | The number of clicks that a student has made on problems this week.
TotalClicksVideo | The number of clicks that a student has made on videos this week.
TotalClicksWeekday | The number of clicks that a student has made on the weekdays.
TotalClicksWeekend | The number of clicks that a student has made on the weekends.
TotalTimeProblem | The total (cumulative) time that a student has spent on problem events.
TotalTimeVideo | The total (cumulative) time that a student has spent on video events.
Control | TotalClicksVideoLoad | The number of times a student loaded a video.
TotalClicksVideo | The number of times a student clicked on a video (load, pause, play, forward).
AvgWatchedWeeklyProp | The ratio of videos watched over the number of videos available.
StdWatchedWeeklyProp | The standard deviation of videos watched over the number of videos available.
AvgReplayedWeeklyProp | The ratio of videos replayed over the number of videos available.
StdReplayedWeeklyProp | The standard deviation of videos replayed over the number of videos available.
AvgInterruptedWeeklyProp | The ratio of videos interrupted over the number of videos available.
StdInterruptedWeeklyProp | The standard deviation of videos interrupted over the number of videos available.
FrequencyEventVideo | The frequency between every Video action and the following action.
FrequencyEventLoad | The frequency between every Video.Load action and the following action.
FrequencyEventPlay | The frequency between every Video.Play action and the following action.
FrequencyEventPause | The frequency between every Video.Pause action and the following action.
FrequencyEventStop | The frequency between every Video.Stop action and the following action.
FrequencyEventSeekBackward | The frequency between every Video.SeekBackward action and the following action.
FrequencyEventSeekForward | The frequency between every Video.SeekForward action and the following action.
FrequencyEventSpeedChange | The frequency between every Video.SpeedChange action and the following action.
AvgSeekLength | The student’s average seek length (seconds).
StdSeekLength | The student’s standard deviation for seek length (seconds).
AvgPauseDuration | The student’s average pause duration (seconds).
StdPauseDuration | The student’s standard deviation for pause duration (seconds).
AvgTimeSpeedingUp | The student’s average time using Video.SeekForward actions (seconds).
StdTimeSpeedingUp | The student’s standard deviation of time using Video.SeekForward actions (seconds).
Participation | CompetencyStrength | The extent to which a student passes a quiz getting the maximum grade with few attempts.
CompetencyAlignment | The number of problems this week that the student has passed.
CompetencyAnticipation | The extent to which the student approaches a quiz provided in subsequent weeks.
ContentAlignment | The number of videos this week that have been watched by the student.
ContentAnticipation | The number of videos covered by the student from those that are in subsequent weeks.
StudentSpeed | The average time passed between two consecutive attempts for the same quiz.
StudentShape | The extent to which the student receives the maximum quiz grade on the first attempt.

Table 3. Descriptions of 45 behavioral features used in model training as
discussed in Sec. 2.2.2.

##  Appendix A Hyperparameter Tuning

This appendix section details the hyperparameter tuning decisions made in the
model optimization protocol (Sec. 3.2), provided for community replication.
The batch size was set to $64$. We used the Adam optimizer, with an initial
learning rate of $0.001$. For all models, the number of _BiLSTM_ layers and
the number of units per _BiLSTM_ layer were optimized via grid search. For the
BSM model, we extended the grid search to the number of _Dense_ layers and the
number of hidden units per layer in the final cascade. For both the BSM model
and the BTM models, the grid search was further extended to the meta features
to be included in the model (all combinations of meta features) and to the
size of the word embeddings for the Title, the Short Description, and the Long
Description. Details about hyperparameter grids are reported in our code
repository999https://github.com/epfl-ml4ed/meta-transfer-learning. The
selected hyperparameter combination is the one achieving the highest balanced
accuracy (BAC) on a validation set composed of students in
$S_{val}=\cup\;\tilde{S}^{c}\;\forall\;c\in C_{trn}$, with $\tilde{S}^{c}$
denoting $10\%$ randomly selected students for a given course $c$ (we
stratified this selection on course and pass-fail label).

##  Appendix B Features

This appendix section will describe the behavioral features used in model
training in more detail, as discussed in Sec. 2.2.2. Our four selected feature
sets (Regularity, Engagement, Control, and Participation) are highlighted by
previous work [17] as strongly performant feature sets for MOOCs. In total, we
have extracted the 45 features listed in Table 3. The majority of the features
come from the Control and Engagement feature sets. For the mathematical
equations, we refer to the papers in which the features were initially
introduced [16, 31, 11, 17]. A detailed implementation of all features can be
found in EPFL ML4ED’s code repository101010https://github.com/epfl-
ml4ed/flipped-classroom.