# The Sequence Matters in Learning - A Systematic Literature Review

Manuel Valle Torre<EMAIL_ADDRESS>0000-0002-0456-8360 Delft
Institute of TechnologyVan Mourik Broekmanweg 6DelftNetherlands2628 XE ,
Catharine Oertel<EMAIL_ADDRESS>0000-0002-8273-0132 Delft
Institute of TechnologyVan Mourik Broekmanweg 6DelftNetherlands2628 XE and
Marcus Specht<EMAIL_ADDRESS>0000-0002-6086-8480 Delft Institute of
TechnologyVan Mourik Broekmanweg 6DelftNetherlands2628 XE

(2024)

###### Abstract.

Describing and analysing learner behaviour using sequential data and analysis
is becoming more and more popular in Learning Analytics. Nevertheless, we
found a variety of definitions of learning sequences, as well as choices
regarding data aggregation and the methods implemented for analysis.
Furthermore, sequences are used to study different educational settings and
serve as a base for various interventions. In this literature review, the
authors aim to generate an overview of these aspects to describe the current
state of using sequence analysis in educational support and learning
analytics. The 74 included articles were selected based on the criteria that
they conduct empirical research on an educational environment using sequences
of learning actions as the main focus of their analysis. The results enable us
to highlight different learning tasks where sequences are analysed, identify
data mapping strategies for different types of sequence actions, differentiate
techniques based on purpose and scope, and identify educational interventions
based on the outcomes of sequence analysis.

Learning Analytics, Learning Sequences, Sequence Analysis, Literature Review

††copyright: none††journalyear: 2024††copyright: rightsretained††conference:
The 14th Learning Analytics and Knowledge Conference; March 18–22, 2024;
Kyoto, Japan††booktitle: The 14th Learning Analytics and Knowledge Conference
(LAK ’24), March 18–22, 2024, Kyoto, Japan††doi:
10.1145/3636555.3636880††isbn: 979-8-4007-1618-8/24/03††submissionid:
178††ccs: Applied computing Interactive learning environments††ccs: General
and reference Surveys and overviews††ccs: Applied computing Computer-assisted
instruction

This version is for personal use and not for redistribution. The final version
was published as part of the proceedings of the 14th Learning Analytics and
Knowledge Conference (LAK ’24). March 18–22, 2024, Kyoto, Japan.
https://doi.org/10.1145/3636555.3636880

##  1\. Introduction

Educational Technologies (EdTech) have rapidly scaled teachers’ capacity for
instruction, for example through live, virtual and recorded lectures (El
Aouifi et al., 2021; Mubarak et al., 2021). EdTech, has also increased the
opportunity for learners to actively engage with learning materials, from
practice problems in electronic environments to automated assessments, or
annotation tools (Dever et al., 2022; Li et al., 2022). However, some
processes that link teaching and learning in a continuous cycle, such as
formative assessment, and personalized support and feedback, have not scaled
at the same rate (Sedrakyan et al., 2020). From its origins, Learning
Analytics (LA) intends to close this cycle using data generated by learners in
educational systems, and processing it to obtain insights grounded in learning
theory and provide (semi) automated learning support and feedback (Clow,
2012).

Existing implementations of LA, such as dashboards or nudges, are often based
on descriptive analytics of performance metrics (Valle Torre et al., 2020),
designed for summative assessment and (self) monitoring (Wong et al., 2019).
However, analytics of the learning process with the detail level to enable
timely assessment and support are an ongoing development (Sedrakyan et al.,
2020). Learning is a process that unfolds and changes over time, and
aggregating or counting the actions the learner executes is limited in its
narrative power (Saint et al., 2020). For that reason, it is critical to
understand the execution of learning tasks as a temporal and sequential notion
(Molenaar and Wise, 2022). Using sequential data to observe, understand, and
support learning is not exclusive to LA, but has a history in fields such as
Educational Data Mining (EDM), Intelligent Tutoring Systems (ITS), and
Artificial Intelligence in Education (AIED) (Wang et al., 2023). This results
in a lack of common definitions of concepts such as learning sequences or
sequence units. Furthermore, there are no established best practices of the
analysis methods that can be applied for specific purposes, educational
settings or scenarios (Knight et al., 2017). As a consequence, research is
often in the exploratory stage, using data from scenarios where the analysed
course or task is over, without intervention (Motz et al., 2023).

In this article, we work towards a broader understanding of the variety of
definitions, data points, methods, and interpretations used for the analysis
of learning as a sequence. We aim to enable a broader understanding of the
learner context to be taken into account for intelligent feedback and
analytics interventions. In the following, we will first introduce the state-
of-the-art of learning sequence analytics and then present the analysis of 74
articles considering the given dimensions.

##  2\. Sequence Analysis and Learning Sequences

Sequence Analysis (SA) of human behaviour, where a sequence is an ordered list
of events, allows the contextualization of these events, their relationships
with each other, and the effects of part of or the whole sequence (Abbott,
1995). SA is especially useful to research phenomena that occur as a process
across time, where the use of static attributes and frequency rates may not be
informative enough to understand their development (Abbott, 1995). The
analysed sequences can include actions executed by a person as well as
reactions of others in their environment, i.e. a peer or an intelligent agent.

Sequences can have different interpretations in education, in this review, we
focus only on the sequences of actions that learners execute during a learning
task. For example, the steps followed to solve a problem (Wang et al., 2023)
or the sequences of answers that learners provide in guided practice
environments, and how certain sequences may relate to learners’ behaviours
(Venant et al., 2017) or characteristics (Baker et al., 2010).

###  2.1. Sequences of Actions in Learning Tasks

Learning tasks refer to situations when learners actively engage with the
learning materials which require them to integrate domain-specific knowledge
and procedures with more general critical thinking and analysis skills (Nokes
et al., 2010), such as solving complex problems (Kirschner and van
Merrienboer, 2008) or scientific inquiry tasks (Wang et al., 2023). For tasks
in different domains, there are behavioural patterns that appear with
different levels of practice. For example, experts in a domain may show deeper
categorization of a problem, use forward-thinking strategies in their
execution of the solution or iteratively evaluate intermediate solutions
(Nokes et al., 2010). In these tasks, learners have a large number of
operators or actions available in the problem space, or learning environment,
making the combinations leading to a solution difficult to observe.
Furthermore, defining and describing such combinations is not trivial, since
they are ”automated”, or stored in one’s memory as procedural learning, making
it difficult to explain by experienced individuals (Nokes et al., 2010). As a
consequence, the assessment and monitoring of learners are particularly
difficult for teachers, especially considering the scale provided by
educational technologies and learning platforms (Hurtig et al., 2022b). The
data produced by these systems, however, can be leveraged to scale the
capabilities of teachers, by analysing sequences of learner’s data records to
identify behaviours and reveal learning processes (Matcha et al., 2019).

###  2.2. Trace Data Sequences and Analysis

The reliability of trace data comes from its ability to express dynamic
events, more than as aggregated, static aspects of a measure (Wang et al.,
2023). Using analytics to find patterns in the sequences that indicate
specific tactics or common issues can help practitioners obtain insights into
the use of different strategies. Such patterns can be a set of learning
actions that occur in succession for a proportion of the learners, from
partial segments to complete sequences. In addition, given that the patterns
are extracted from sequences of actions, they can be interpreted by the
teacher and even learners themselves (Tang et al., 2021a). For example, in a
3D geometry game where learners can insert, remove, and rotate shapes to solve
puzzles, teachers cannot assess nor give feedback to learners based on the
gameplay records without assistance. Common patterns that lead to a failed
assignment can be identified as mistakes and teachers can prioritize which to
address (Gomez et al., 2021).

Analysing sequences of trace data throughout a learning task offers practical
advantages as well (Taub and Azevedo, 2019): when systems process data traces
as sequences of learner actions, there is a constant input of information to
incrementally update the learner model, enabling the system with the
flexibility to constantly adapt to the learner’s level or intervene whenever
necessary (Corbeil et al., 2020). This allows practitioners, and eventually,
an intelligent system, to predict if the current path is trending towards an
undesirable outcome, to determine which action patterns are leading to that
result and how to leverage these insights to optimise learning (Tang et al.,
2021b; Riofrío-Luzcando and Ramírez, 2016).

###  2.3. Related Work

The analysis of learning sequences is emerging from several fields, and while
there are no surveys specifically on the topic, there are related works from
the EDM, LA and AIED communities. Regarding the use of educational data to
observe learner behaviour and predict performance, Xiao et al. (Xiao et al.,
2022) analyse EDM methods in the age of big data to discover the trends around
the complete data mining pipeline, such as data collection and preprocessing,
feature engineering, classifier and ensemble method selection, and
interpretation of results. While they mention techniques that analyse
sequential data, there are no definitions of learning sequences or sequence
units. On the other hand, the survey by Bogarín et al. (Bogarín et al., 2018)
focuses on Educational Process Mining (EPM), a field that studies the process-
oriented view of EDM, particularly as an end-to-end process. The authors
describe Virtual Learning Environments (VLE) and the Event Logs they produce
as valuable data sources for sequential analysis, as they are detailed records
of learners’ actions. They mention methods handling learning as sequences,
including Sequential Pattern Mining, but the definitions outside of process
mining are not addressed.

Regarding the use of trace data to observe learners’ actions, the review by
Wang et al. (Wang et al., 2023) analyses research on the use of log data from
Open-Ended Learning Environments (OELEs) to understand scientific inquiry and
problem-solving. Their conceptual framework consists of 4 points: the
competencies measured, mainly problem-solving and inquiry; the observations
used, ranging from multiple-choice questions to log data; the feature
extraction and modelling processes and the interpretation of the analysis.
While the focus is not only on sequential methods, almost half of the 70
reviewed papers use sequential features for analysis, with 29 applying pattern
mining to identify behaviours. The review by Nadimpally et al. (Nadimpalli et
al., 2023) surveys Artificial Intelligence techniques for adaptive learning
systems, identifying two main types: to model learner behaviour and to
organize learning content. Concerning modelling learner behaviour, they found
that the most common methods are Similarity-based Pattern Recognition,
Probabilistic Graphical Models, and Deep Learning: all of them capable of
processing sequential data. Existing literature explores the growing use of
temporal data to identify, classify or predict learner behaviours, which often
involves sequence analysis. However, a focused study of when to use sequence
analysis, the data units that can be leveraged and which methods to implement,
is less covered.

##  3\. Research Questions

Addressing the shortcomings from the previous section, this study intends to
establish a common ground for Sequence Learning Analytics from the literature,
so researchers can use this to find similar works, compare their approach to
the state-of-the-art, and leverage insights of existing work, allowing the
community to also focus on the intervention design. In this review, a learning
sequence is defined as an ordered list of context-specific actions, with each
unit being a systematically mapped learning action from one or more data
traces. The following questions guide our analysis:

  1. (1)

How are learning sequences analyzed to describe and support learners’ actions
in computer systems?

    1. (a)

Which types of learning tasks are analysed as sequences?

    2. (b)

What is the motivation or purpose when using learning sequences?

    3. (c)

Which actions are used as the learning sequence units?

    4. (d)

Which methods are used to analyse learning sequences? And what are the
interpretations of the patterns that result from them?

    5. (e)

Which educational interventions are designed with the obtained insights? Which
learning principles are used to support and evaluate them?

##  4\. Method

In this review, we followed the Preferred Reporting Items for Systematic
reviews and Meta-Analyses (PRISMA) (Page et al., 2021), to answer the
questions above, in Section 3. The queries, information sources and
eligibility criteria are described in this section, followed by the document
collection and inclusion process, and the data extraction method.

The queries were executed in SCOPUS and Web of Science (WoS), for the domains
of Educational Data Mining (EDM), Artificial Intelligence in Education (AIED),
Learning Analytics (LA), Intelligent Tutors and Learner Modeling, exploring
the fields of title, abstract and author-defined keywords. SCOPUS and WoS were
selected as they include most of the top publications in such domains, such as
the proceedings of the conferences on Learning Analytics and Knowledge,
Educational Data Mining, and the International Conference on Computers in
Education, as well as the Journal of Learning Analytics, British Journal of
Educational Technology, Journal of Educational Data Mining, and Computers and
Education: Artificial Intelligence. The queries were executed on September
2023, including articles from 2010 to 2023, and they focus on the use of
sequential analysis in an educational setting, where the obtained patterns are
leveraged for an educational intervention. Two queries were executed, one per
database, using the query below, with the results described in Section 4.2.

(”data mining” OR ”artificial intelligence” OR ”learning analytics” OR
”intelligent tutor*” OR ”learner model*”) AND  ( sequen* OR procedur* OR
series ) AND ( educatio* OR student OR teacher OR instructor ) AND ( feedback
OR assessment OR predict* )

###  4.1. Study Selection Process and Eligibility Criteria:

The results were included if they were full-length articles, published in a
peer-reviewed source, and written in English. This was followed by two rounds
of screening, the first one based on the abstracts and the second on full text
for those that were unclear from the first screening. To be included, the
publication had to be implemented in an educational setting, where the
learning sequences refer to an ordered series of actions during a learning
task. Non-empirical works are excluded, as are works that only mention
education as part of the institution name or a possible use case.
Additionally, publications were excluded if they analysed unrelated
sequencing, like proteins, or where sequences were part of the methodology,
such as “a sequence of trials” or “a series of interviews”. Finally, works
exploring learner patterns across multiple tasks were excluded, such as
navigation patterns on the elements accessed in MOOC or the courses in a
curriculum.

###  4.2. Data Collection Process and Data Items

Figure 1. Document Flow Diagram

[Document selection Flow Diagram]Selection flowchart from 1341 documents to
the final 74

Reference lists were exported from SCOPUS and WoS, and then transferred to
Google Sheets for screening. The reason for exclusion or inclusion was
recorded for each article, and the values are shown in Figure 1. Of the
articles obtained from the queries, 953 were unique, then in the abstract
screening 717 were excluded since they were not in an educational setting, not
empirical work, the use of learner action sequences was not part of the
analysis or did not focus on a single learning task. The full text of the
remaining 236 articles was screened, and 162 were excluded based on the
criteria described above, resulting in 74 articles included. Document
collection and inclusion were performed by the first author, using the
criteria previously defined with the rest of the team.

###  4.3. Analysis Procedure

The included studies were coded by the first author using ATLAS.ti (a
qualitative research software) with the Research Questions from Section 3) as
a guide, following negotiated consensual validation with the rest of the
authors to iteratively review the codes (Sandelowski and Barroso, 2007). For
each question, we determined 3 or 4 possible categories from the related work
in Section 2.3, and then the full text of the articles was reviewed and coded
using the questions and categories. If a code answered one of the questions
but did not fit any of the categories, a new category was created.
Additionally, a second coder reviewed a sample of 10% of the articles and the
results were compared to ensure the reliability of the coding process, which
showed an inter-rater reliability kappa of 0.922, where any differences were
discussed and addressed.

For the first question, we identified the learning tasks considering the
educational construct they are based on: deliberate practice, solving a
problem or scientific inquiry (Wang et al., 2023). The first one refers to
tasks where learners practice a skill through repetition, usually with a
system tracking their progress and adjusting accordingly. The second one
refers to tasks where learners have to solve a complex problem in a virtual
environment, and the third one refers to tasks where learners have to obtain
and aggregate information.

On the other hand, the purpose of sequence analysis can be defined similarly
to the goal of data mining and analytics tasks, such as clustering,
classification or prediction (Xiao et al., 2022), in line with the findings in
(Wang et al., 2023). While all the reviewed works identify patterns using
sequence analysis, some of them go further and compare the patterns between
groups of learners, for example, to identify behaviour differences among
learners who received different experimental treatments, or to classify them
by the use of strategies. Finally, some works use the identified patterns to
predict the next action of a sequence, the success of such action, or a
learner’s state.

Regarding the units of the sequences, the reviewed research uses data sources
such as log files or event records (Winne, 2010), which have to be mapped to
representations of meaningful actions, to be analysed and understood as part
of the learning sequence (Nesbit et al., 2007). These sequence units can
categorised as learning actions, which are domain or system-specific actions
such as selecting and using a system tool in an OELE (Wang et al., 2023), or
metacognitive actions, usually defined based on an educational theory, such as
self-explanation (Kinnebrew and Biswas, 2011). The third category is problem
attempts, the series of submissions in a guided practice system (Xiong et al.,
2016; Nadimpalli et al., 2023).

For the implemented methods, we obtained the categories of methods that use
sequential data from related literature on AI in Education and Process Mining
(Nadimpalli et al., 2023; Bogarín et al., 2018): Probability Modeling, Deep
Learning, Sequential Pattern Mining, Educational Process Mining and Knowledge
Tracing. Additionally, we identify the scope of the resulting sequences as the
reference for interpretation, from system-specific tactics to general
strategies for a complete task (Bogarín et al., 2018).

Regarding interventions and educational theories, we seek to identify which
works go beyond the observation and exploration stage, which is why we
included assessment, prediction and feedback in our query, in Section 4.
However, considering the challenge of implementing and evaluating an
educational intervention (Rienties et al., 2017; Motz et al., 2023), we
classify if articles implement an intervention, or if they explicitly mention
a potential intervention or not. Additionally, the impact of interventions
relies on a strong theoretical foundation (Wise and Shaffer, 2015), so we
identify the educational theories used to support or interpret the results of
sequential analysis (Khalil et al., 2022).

###  4.4. Limitations

The search was limited to Web of Science and SCOPUS, while these libraries
include a wide range of publications, the results are not exhaustive.
Additionally, there are no established practices to mention sequence analysis
in the title, keywords, or abstract, in consequence, some articles may not
have appeared in our queries. Furthermore, abstracts were used to assess the
first round of inclusion, there is the possibility that articles were
erroneously excluded. Regarding the analysis, the articles were coded only by
the first author, however, we addressed this limitation by following
negotiated consensual validation with the author team (Sandelowski and
Barroso, 2007) and assessing the agreement of a sample with a second coder.

##  5\. Results

Our analysis included 74 articles, with the number of publications doubling in
the second half of our search range, suggesting a constant increase in
interest in the field, shown in Figure 2. The articles were analysed following
the Research Questions in Section 3, to establish the trends of the different
types of learning tasks, the different purposes of the sequence analysis, the
types of units and analysis methods used, and any implemented interventions.
The following sections address each of these questions, with the results
available as an open-access dataset (Valle Torre et al., 2023), and as an
interactive interface at https://sequence-dashboard.learn.ewi.tudelft.nl/.

Figure 2. Articles, and Task Types, per Year

[Bar chart of Articles, and Task Types, per Year]Stacked, vertical, bar chart
of the number of articles published per year, aggregated by task types, going
from 2010 to 2023

###  5.1. Which types of learning tasks are analysed using sequences?

This review focuses on tasks that require active engagement from the learner,
which instructional design models such as the 4C/ID model (Kirschner and van
Merrienboer, 2008), consider as authentic problems and practices. However,
they can follow different educational designs. We identify three main types of
tasks: guided practice, problem-solving, and knowledge building; the yearly
number of publications is shown in Figure 2. The first type is guided
practice, characterized by a set of practice questions that are part of a
topic, where a system uses the sequence of answers to constantly measure and
adapt to the learner’s knowledge level. We registered 40 articles of this
type, 2 of them using dialogue (González-Brenes et al., 2011; Myers, 2021),
and 2 as a quiz (Juhaňák et al., 2019; Qu et al., 2019).

The second type of task is problem-solving, a complex task integrating domain
knowledge with domain-specific and general skills. In these tasks, the
learner’s interactions with the system can be interpreted as the operations
executed towards a solution state (Anderson, 2000). While these tasks
generally have a defined objective, they can be satisfied with different
sequences of actions or applied operators, from a wide variety of actions
available within the educational system. The 17 articles using data from a
variety of problem-solving tasks, including programming assignments
(Kesselbacher and Bollin, 2019), mechanical engineering problems (Smith et
al., 2015) or procedures in a biotechnology virtual lab (Riofrío-Luzcando and
Ramírez, 2016; Riofrío-Luzcando et al., 2017).

Finally, some of the tasks require that the learner collects and processes
information, from video lectures or reading materials, and then generates and
evaluates a representation of knowledge, such as a concept map (Bouchet et
al., 2012). In these tasks, the final result may be similar for learners who
followed different strategies, or sequences of steps, and would benefit from
personalized feedback. We identified 17 articles that can be classified as
knowledge-building tasks, for example, where a learner interacts with articles
on climate change and then generates a causal model to explain the
relationships between the learned concepts (Omae et al., 2018).

In summary, we identified 40 guided practice tasks, 17 problem-solving and 17
knowledge-building tasks, with the first type gaining ground in recent years,
shown in Figure 2. This may be due to the introduction of increasingly
sophisticated deep learning methods or the availability of public datasets
(Xiong et al., 2016; Wu and Ling, 2023; Ni et al., 2023), eliminating the need
to design new systems or collect data. Additionally, it might be related to
the research objectives, described below.

###  5.2. What is the motivation or purpose of using learning sequences for
analysis?

There are many affordances to the use of learning sequences and sequence
analysis methods in EDM and LA, such as the ability to extract the sequential
patterns from data (Lundgren, 2022), to identify them as tactics or strategies
(Morshed Fahid et al., 2021), or to include temporal structures in behaviour
analysis (Li et al., 2022). Most articles identify the patterns as a learning
behaviour, tactic or strategy, however, some of them go further and use these
patterns for classification or for prediction. We found 8 cases of pattern
identification, for example, as the self-regulatory processes that occur
before a help-seeking event. These patterns can be used to anticipate the
struggle of learners who may not use help-seeking tools correctly (Lajoie et
al., 2015). Another case of identification is when groups of learners receive
different experimental treatments, such as a feedback strategy (Sun et al.,
2019), and then the patterns of the groups are compared. This means that in
these 6 cases, the identified sequences are used as part of the evaluation of
an educational intervention (Kinnebrew et al., 2014).

Going a step further, in the 24 articles where the purpose of the sequence is
to serve for classification, researchers first separate the learners by a
given characteristic, such as proficiency in the task (Rajendran et al., 2018)
and then identify the patterns per group. In these cases, like in traditional
data mining classification tasks, a small percentage of the data is used to
evaluate the accuracy of the method, or how well can it classify a learner by
their behaviour (Rhienmora et al., 2010; Rajendran et al., 2018; Herold et
al., 2013). In only one article the grouping is done by applying clustering
algorithms on the sequences of all learners, and then samples of the groups
are reviewed and labelled as a certain strategy, such as Efficient Completion
(Morshed Fahid et al., 2021).

Regarding prediction, the purpose is to use a sequence as input but there is a
variety of outputs: in 8 cases it is the learner’s performance, such as
success or failure on the task (Smith et al., 2015; Chu et al., 2021), in 2 is
the learner’s state such as emotional or cognitive states (Corbeil et al.,
2020; Baker et al., 2010) and for 1 it is the next action they might do
(Riofrío-Luzcando and Ramírez, 2016). For those works focusing on prediction,
this is often with the vision of an early warning system or even automated
interventions. Finally, in 24 cases, the purpose is to predict the success of
a learner’s next action, often to determine their knowledge level for a given
topic (Vie and Kashima, 2019).

###  5.3. Which actions are used as the units of learning sequences?

In the analysed articles we found a broad variety of data records logged,
including clicks on a web interface, sensor data from haptic devices, and text
input in command line interfaces. A single case uses the data as-is, as the
authors consider them detailed enough for the sensor data collected from a
dentistry virtual reality simulator (Rhienmora et al., 2010). In the rest of
the publications, we found three different types of combining data records
into traces: learning actions, problem attempts, and meta-cognitive actions.
We identified 26 articles that use learning actions as units, which usually
represent the different ways that learners apply operators in problem-solving
tasks, defined in Section 5.1, such as editing code, executing a command, or
using a tool. Mapping the data records into such actions allows researchers,
teachers and students to easily analyse and interpret the results of sequence
methods. The downside is that this mapping usually requires expertise with the
domain and system, to know which actions are possible, and it also limits the
possibility to generalize the analysis and results.

In the second case, identified in 37 of the reviewed works, the collected data
records are better portrayed as problem attempts because they are always
delimited by a submission event: when a student submits a response to be
evaluated by the system, usually named Intelligent Tutoring System (ITS), and
they correspond to those using the guided task type. Considering that there is
a clear definition of where one unit starts and ends, there is little focus on
the process of mapping data records to units.

Finally, for the 10 articles using meta-cognitive actions, we found that they
introduced a conceptual background from different learning theories, and then
use the selected theory to construct the sequence units. Eight of them use a
Self-Regulated Learning (SRL) model to map certain records into a meta-
cognitive action. From the rest, one maps the data to meta-cognitive states,
which they refer to as on-task and off-task states (Sun et al., 2019), and the
other to a variety of meta-cognitive and cognitive behaviours, such as
evaluating, help-seeking, and skipping video (Chen et al., 2022). Using
conceptually mapped actions reduces the granularity and length of the
sequences to be analysed since they usually aggregate data records into just a
few possible options, compared with the simpler learning actions above.
Additionally, this means less variability during analysis and a more direct
interpretation.

The sequence units used for analysis are strongly related to the type of
learning task, the 26 articles with learning actions tend to be on problem-
solving, using specific actions in specific problems, and the 37 using problem
attempts are generally on guided practice tasks, focusing more on the accuracy
of the learner’s model. Finally, the 10 cases of meta-cognitive actions are
usually in the knowledge-building tasks, focusing more on self-regulated
learning processes.

###  5.4. Which methods are used for the analysis of learning sequences?

The methods in the reviewed articles usually take a set of sequences as input,
where each one corresponds to learning actions executed by one learner during
a task. The methods that we found range from the manual analysis of game logs
(Owen et al., 2016) to deep learning models capable of predicting the
performance of the learner (Smith et al., 2015), with many others among them.
In our analysis, we classify the different methods into 5 categories:
Sequential Probability Models, Educational Process Mining, Deep Learning,
Knowledge Tracing, and Sequential Pattern Mining (SPM).

Scope |  Method Category |  Analysis Method |  References
---|---|---|---
Macro-level Patterns |  Educational Process Mining |  Fuzzy mining, PetriNet, Process Maps |  (Rajendran et al., 2018; Neyem et al., 2017; Juhaňák et al., 2019)
Other Methods |  N-gram processing, Network Visualization |  (Lundgren, 2022; Hurtig et al., 2022b, a)
Sequential Probability Model |  Markov Chain Model, Markov Chain Process, Hidden Markov Model, Microsoft Sequence Clustering, Markov Logic Networks, Probabilistic Automata and Hidden Conditional Random Fields |  (Dever et al., 2023; Rhienmora et al., 2010; Ha et al., 2011; Meliana and Nurjanah, 2018; Omae et al., 2018; González-Brenes et al., 2011; Kinnebrew and Biswas, 2011; Tang et al., 2021a; Dever et al., 2022; Riofrío-Luzcando et al., 2017; Riofrío-Luzcando and Ramírez, 2016)
Meso-level Patterns |  Deep Learning |  GRU, MLP, CNN, LSTM, SAE |  (Chu et al., 2021; Smith et al., 2015; El Aouifi et al., 2021; Qu et al., 2019; Mubarak et al., 2021; Geden et al., 2020; Corbeil et al., 2020; Zhou et al., 2022)
Knowledge Tracing |  PFA, ModPFA, BKT, DKT, DKVMN, AKT, and many more |  (Banno et al., 2020; Tuti et al., 2019; Gong et al., 2010; Guo et al., 2023; Zhang et al., 2023; Liu et al., 2023; Mandlazi et al., 2021; Wu and Ling, 2023; Xiong et al., 2016; Ni et al., 2023; Lu et al., 2023; He et al., 2020; Vie and Kashima, 2019; Jiang et al., 2022; Gan et al., 2020; Xia et al., 2023; Eagle et al., 2018; Sahebi et al., 2014; Schatten and Schmidt-Thieme, 2016; Sahebi et al., 2016; Geng et al., 2023; Yang et al., 2022; Cook et al., 2017)
Other Methods |  Dynamic Time Warping, Model-free bounded optimization, Course-agnostic Student Performance Model |  (Morshed Fahid et al., 2021; Chen et al., 2018; Schmucker and Mitchell, 2022)
Micro-level Patterns |  Sequential Pattern Mining |  SPAM, Differential, Temporal Interestingness of Patterns in Sequences (D-TIPS), Pex-SPAM, Generalized Sequential Pattern Mining, Frequent Item set, |  (Kinnebrew et al., 2014; Gomez et al., 2021; Myers, 2021; Akhuseyinoglu and Brusilovsky, 2021; Wang et al., 2019; Gomez et al., 2020; Taub and Azevedo, 2019; Norm Lien et al., 2020; Bouchet et al., 2012; Lin, J. et al., 2022; Herold et al., 2013; Venant et al., 2017; Latypova, 2022; Akhuseyinoglu and Brusilovsky, 2022)
Manual Analysis |  Manual Labeling, Sequence of Action Tables, State Permutations |  (Duong et al., 2013; Baker et al., 2010; Kesselbacher and Bollin, 2019; Sun et al., 2019)
Other Methods |  Visualization, Lag Sequential Analysis, GSEQ (Generalized Sequential Querier), Time-embedded n-gram |  (Supianto and Hafis, 2019; Liu et al., 2018; Li et al., 2022; Chen et al., 2022)
Table 1. Methods per Category and Scope

Sequential Probability Models estimate the probabilities of transition between
elements in a sequence, based on the previous one or more elements (Dever et
al., 2023). On the other hand, Educational Process Mining refers to the EDM
technique to discover, visualise and compare process models, focusing on the
complete sequence (Rajendran et al., 2018). Deep Learning methods use
Artificial Neural Networks (ANN) to predict the outcome of a sequence, mainly
Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM) networks, as
they are better suited to handle sequential data than other ANNs (Chu et al.,
2021). While Knowledge Tracing approaches leverage probabilistic, machine
learning or deep learning techniques, their main focus is to model the
knowledge evolution of a learner throughout their interaction with an
educational system (Guo et al., 2023). Sequential Pattern Mining methods refer
to a group of data mining techniques used to find statistically significant
patterns—or subsequences—in sequence datasets, such as actions that happen
frequently in the same order across learners’ records (Kinnebrew et al.,
2014). Manual Analysis methods are often used in combination with other
methods but are also used on their own in a few cases, for instance by
reviewing and classifying the actions of individual learners in a small group
(Baker et al., 2010). These methods often relate to the scope of the analysis,
from the full-task strategies shown by a learner population to short tactics
that certain students apply depending on context: the Macro-level Patterns
identify strategies that span across the complete task and are analysed for a
population or group. The Meso-level refers to behaviours that appear in large
part of the task, but the focus is on the individual level. Finally, the
Micro-level patterns are short tactics that occur in a particular context.
Shown in Table 1, we group the examples of each method and the scope that they
belong to.

###  5.5. Educational Interventions - What is the final intervention designed
using the obtained insights?

Educational interventions provide students with the support required to
acquire the skills and knowledge being taught. However, interventions remain
one of the main challenges in the learning analytics cycle (Rienties et al.,
2017). This is reflected in our findings, where most reviewed publications are
exploratory studies with potential interventions, for instance, as an early
warning system that can notify teachers about learners using tactics that lead
to poor performance (Wang et al., 2019), or automated systems that can detect
if a learner might go into a blocking state (Corbeil et al., 2020). There were
also 5 prototype teacher dashboards (Juhaňák et al., 2019; Gomez et al., 2020,
2021; Supianto and Hafis, 2019; Bryfczynski et al., 2013), with one of them
receiving a positive response from a usability survey with 56 participants
(Hurtig et al., 2022a).

In our review, we found 2 articles that design and implement an intervention.
In the first, the authors integrate the patterns obtained to provide
suggestions to students in a programming tutoring system. The suggestions
included reading the manual or asking a peer for help when an error state is
identified, to nudge the learners into a self-regulation strategy, for
example, reflection-then-success after a sequence that could lead to failure
(Venant et al., 2017). The impact of this intervention, however, is not
evaluated. The second one implemented and evaluated the use of learning
sequences, by creating a “good-learner path” model using a Markov Decision
Process in a guided practice system, so it can automatically generate the
content sequence for first-time users. The model was then evaluated by
comparing the test results of learners who followed the created path and
learners who were provided with exercises one by one based on their prior
results. Their approach produced 19.5% more good learners and 20.5% lower
standard deviation than the control group (Meliana and Nurjanah, 2018).

##  6\. Discussion

###  6.1. Learning Tasks and Analysis Purpose

Regarding the purpose of sequence analysis, the common objective is to extract
patterns from the data and identify the real-world behaviours they describe.
In some cases, these patterns are then used to classify the learner between
groups or to predict the learner’s state, their subsequent action, or the
outcome of the task. In most research using problem-solving tasks, patterns
are closely related to the scenario, for example, how novices implement
functions in programming. A few cases use these scenario-specific patterns to
understand more general skills, for instance, identifying if learners follow
linear or iterative processes for problem-solving. Similarly, for the
knowledge-building tasks, the focus is to find information processing
strategies and the meta-cognitive processes executed by the learners
(Kinnebrew and Biswas, 2011; Kinnebrew et al., 2014). Using an educational
lens to determine the purpose of a study, such as complex problem-solving or
self-regulation, facilitates the generalisation of the methods and results,
regardless of the task.

On the other hand, the purpose of many guided practice tasks is to improve the
prediction of the learners’ performance, often to evaluate a proposed method
against the state-of-the-art, as is the case for most of the 16 articles using
the ASSISTments111https://sites.google.com/site/assistmentsdata/datasets
dataset. In these studies, the sequential patterns receive less attention, and
educational theory is leveraged in the prediction function, such as learning
and forgetting, or while calculating the difficulty of a learning component.
While this field has seen important growth with the popularity of Neural
Networks (see figure 2), the real-world application and explainability of
modern techniques are still key challenges (Song et al., 2022).

###  6.2. Learning Actions and Methods

From our analysis, there seem to be increasing similarities in how researchers
of different domains use sequences for the analysis of learner behaviour. For
example, we found that 25 out of the 37 articles using Problem Attempts as the
units of sequence, also integrate contextual information about the learner and
their actions before submitting a problem, such as temporal features or their
previous knowledge. Another common case is the use of statistical methods in
combination with sequential methods, such as using logistic regression with
features extracted from a sequential technique. Furthermore, researchers often
implement more than one sequence analysis technique, using one to find case-
specific action units, and then another to explore how these actions are
sequenced in a full task strategy (Tang et al., 2021a). This may be a
consequence of recent efforts towards the transparency and explainability of
data-driven analysis, as well as the increasing accessibility of technology
and analysis techniques.

The use of machine learning and deep learning methods to power automated or
semi-automated decisions has recently started to see widespread
implementation. This may be due to the recent efforts to mitigate risks
associated with AI, for instance by pursuing transparency, causality, privacy,
fairness, trust, usability, and reliability; some of the goals of Explainable
AI (XAI) (Lu et al., 2023). Explainability is especially important when
applied to education, as the explanations can also have an educational
component for learners, and it can aid teachers in their assessment of
learners and their courses (Lu et al., 2023). Regarding accessibility,
different techniques are available via tools and libraries, we found several
articles that use the Waikato Environment for Knowledge Analysis
(Weka222https://www.cs.waikato.ac.nz/ml/weka/) for multiple machine learning
classifiers and predictors, (Herold et al., 2013; Myers, 2021; Sahebi et al.,
2014; Riofrío-Luzcando et al., 2017; Owen et al., 2016; Padrón-Rivera and
Rebolledo-Mendez, 2015; Bouchet et al., 2012), 2 more that apply Sequential
Pattern Mining with TraMineR for R333http://traminer.unige.ch/ (Lin, J. et
al., 2022; Tuti et al., 2019), as well as Markov Models via MeTA
Toolkit444https://github.com/meta-toolkit/meta (Tang et al., 2021b), and the
Microsoft Sequence Clustering Algorithm (Riofrío-Luzcando et al., 2017). It is
important to note that these tools were explicitly mentioned in their
respective articles, further facilitating reproducibility and comparison. This
allows researchers to work with known methods throughout their analysis
process, without having to develop, test, and evaluate them from scratch.
Finally, we provide the dataset (Valle Torre et al., 2023) of the reviewed
works as an interactive interface 555https://sequence-
dashboard.learn.ewi.tudelft.nl/ where researchers can filter the methods
depending on the scope of their study, as shown in Table 1.

###  6.3. Interventions

Some of the reviewed works mention potential interventions that can be
designed using the obtained patterns. For example, adaptive scaffolding in
problem-solving tasks, personalized feedback for a knowledge-building system,
or detection of the learners’ level in guided practice. However, only 2
implemented an intervention, with 1 of them evaluating it, while 20 articles
had no mention of a potential intervention at all. This is in line with recent
publications such as ”LAK of Direction”, where they found that 11% of their
reviewed articles introduced an intervention into a learning environment (Motz
et al., 2023).

The use of learning principles, such as cognitive load (Myers, 2021) or
learning and forgetting curves (Gan et al., 2020), is often present when
describing or framing the problem. Very few use learning principles in the
interpretation of patterns or interventions. This may be due to the focus on
task- or system-specific tactics more than general learning behaviours.
However, actionable results in LA research require a strong theoretical
foundation, present in the analysis and the interpretation of the findings
(Wise and Shaffer, 2015). The theories mentioned in the framing of the problem
should remain through the complete process, such as the use of Neo-Piagetian
theory to classify patterns as programming strategies related to expertise
(Kesselbacher and Bollin, 2019). This was the case for the 10 articles where
the purpose is to identify meta-cognitive strategies, with 8 of them using a
Self-Regulated Learning model to map the data records into activities,
interpret the patterns after analysis, and describe a potential intervention
(Khalil et al., 2022).

##  7\. Conclusion

In this document, we reviewed the different learning tasks, research purposes,
action units, and methods of learning sequence analysis in educational
systems, as well as the interventions that researchers design and propose with
the obtained insights. We described the major trends that can help to
establish an overview of the use of Learning Sequence Analysis, aggregated in
an open repository (Valle Torre et al., 2023) and an interactive interface
available in https://sequence-dashboard.learn.ewi.tudelft.nl/. This is to
assist researchers in finding, comparing, and using other’s efforts as a
foundation and move past the exploratory stage; as noted in one of the
reviewed articles (Riofrío-Luzcando and Ramírez, 2016), ”the community is
missing more generally applicable results.”

###### Acknowledgements.

This project has been funded by the Leiden-Delft-Erasmus Centre for Education
and Learning (LDE-CEL). The authors thank the second coder, Rebecca Glans.

## References

  * (1)
  * Abbott (1995) Andrew Abbott. 1995.  Sequence Analysis: New Methods for Old Ideas.  _Annual Review of Sociology_ 21 (1995), 93–113.  arXiv:2083405
  * Akhuseyinoglu and Brusilovsky (2021) Kamil Akhuseyinoglu and Peter Brusilovsky. 2021.  Data-Driven Modeling of Learners’ Individual Differences for Predicting Engagement and Success in Online Learning. In _Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization_ _(UMAP ’21)_. Association for Computing Machinery, New York, NY, USA, 201–212.  https://doi.org/10.1145/3450613.3456834
  * Akhuseyinoglu and Brusilovsky (2022) Kamil Akhuseyinoglu and Peter Brusilovsky. 2022.  Exploring Behavioral Patterns for Data-Driven Modeling of Learners’ Individual Differences.  _Frontiers in Artificial Intelligence_ 5 (Feb. 2022), 807320.  https://doi.org/10.3389/frai.2022.807320
  * Anderson (2000) John R. Anderson. 2000.  _Cognitive Psychology and Its Implications, 5th Ed_.  Worth Publishers, New York, NY, US. xvi, 531 pages.
  * Baker et al. (2010) Ryan S.J.d. Baker, Antonija Mitrović, and Moffat Mathews. 2010.  Detecting Gaming the System in Constraint-Based Tutors.  _Lecture Notes in Computer Science_ 6075 LNCS (2010), 267–278.  https://doi.org/10.1007/978-3-642-13470-8_25
  * Banno et al. (2020) Samara Banno, Hayder Murad, and Muntadher Sallal. 2020.  Automated Cognitive Analyses for Intelligent Tutoring Systems. In _2020 IEEE/ACM International Conference on Big Data Computing, Applications and Technologies (BDCAT)_. Leicester, UK, 171–178.  https://doi.org/10.1109/BDCAT50828.2020.00007
  * Bogarín et al. (2018) Alejandro Bogarín, Rebeca Cerezo, and Cristóbal Romero. 2018.  A Survey on Educational Process Mining.  _WIREs Data Mining and Knowledge Discovery_ 8, 1 (2018), e1230.  https://doi.org/10.1002/widm.1230
  * Bouchet et al. (2012) Francois Bouchet, Roger Azevedo, John S. Kinnebrew, and Gautam Biswas. 2012.  Identifying Students’ Characteristic Learning Behaviors in an Intelligent Tutoring System Fostering Self-Regulated Learning. In _International Educational Data Mining Society_. International Educational Data Mining Society, Chania, Greece.
  * Bryfczynski et al. (2013) Sam Bryfczynski, Roy P. Pargas, Melanie M. Cooper, Michael Klymkowsky, and Brian C. Dean. 2013.  Teaching Data Structures with beSocratic. In _Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education_ _(ITiCSE ’13)_. Association for Computing Machinery, New York, NY, USA, 105–110.  https://doi.org/10.1145/2462476.2465583
  * Chen et al. (2018) Binglin Chen, Matthew West, and Craig Zilles. 2018.  Towards a Model-Free Estimate of the Limits to Student Modeling Accuracy. In _Proceedings of the 11th International Conference on Educational Data Mining_ , Vol. 11. Raleigh, NC, USA.
  * Chen et al. (2022) Hsi-Min Chen, Bao-An Nguyen, Chyi-Ren Dow, Nien-Lin Hsueh, and An-Chi Liu. 2022.  Exploring Time-Related Micro-Behavioral Patterns in a Python Programming Online Course.  _Journal of Information Science & Engineering_ 38, 6 (2022).
  * Chu et al. (2021) Yun-Wei Chu, Elizabeth Tenorio, Laura Cruz, Kerrie Douglas, Andrew S. Lan, and Christopher G. Brinton. 2021.  Click-Based Student Performance Prediction: A Clustering Guided Meta-Learning Approach. In _2021 IEEE International Conference on Big Data (Big Data)_. Orlando, FL, USA, 1389–1398.  https://doi.org/10.1109/BigData52589.2021.9671729
  * Clow (2012) Doug Clow. 2012.  The Learning Analytics Cycle: Closing the Loop Effectively. In _Proceedings of the 2nd International Conference on Learning Analytics and Knowledge_ _(LAK ’12)_. Association for Computing Machinery, New York, NY, USA, 134–138.  https://doi.org/10.1145/2330601.2330636
  * Cook et al. (2017) Joshua Cook, Collin F. Lynch, Andrew G. Hicks, and Behrooz Mostafavi. 2017.  Task and Timing: Separating Procedural and Tactical Knowledge in Student Models. In _Proceedings of the 10th International Conference on Educational Data Mining_. International Educational Data Mining Society, Wuhan, China.
  * Corbeil et al. (2020) Jean-Philippe Corbeil, Michel Gagnon, and Philippe R. Richard. 2020.  Probabilistic Approaches to Detect Blocking States in Intelligent Tutoring System. In _Intelligent Tutoring Systems: 16th International Conference, ITS 2020, Athens, Greece, June 8–12, 2020, Proceedings_. Springer-Verlag, Berlin, Heidelberg, 79–88.  https://doi.org/10.1007/978-3-030-49663-0_11
  * Dever et al. (2022) Daryn A. Dever, Nathan A. Sonnenfeld, Megan D. Wiedbusch, and Roger Azevedo. 2022.  Pedagogical Agent Support and Its Relationship to Learners’ Self-Regulated Learning Strategy Use with an Intelligent Tutoring System. In _Proceedings of Artificial Intelligence in Education: 23rd International Conference, AIED_ (Durham, United Kingdom). Springer-Verlag, 332–343.  https://doi.org/10.1007/978-3-031-11644-5_27
  * Dever et al. (2023) Daryn A. Dever, Nathan A. Sonnenfeld, Megan D. Wiedbusch, S. Grace Schmorrow, Mary Jean Amon, and Roger Azevedo. 2023.  A Complex Systems Approach to Analyzing Pedagogical Agents’ Scaffolding of Self-Regulated Learning within an Intelligent Tutoring System.  _Metacognition and Learning_ (May 2023).  https://doi.org/10.1007/s11409-023-09346-x
  * Duong et al. (2013) H.D. Duong, L. Zhu, Y. Wang, and N.T. Heffernan. 2013.  A Prediction Model Uses the Sequence of Attempts and Hints to Better Predict Knowledge: Better to Attempt the Problem First, Rather than Ask for a Hint. In _Proceedings of the 6th International Conference on Educational Data Mining, EDM_.
  * Eagle et al. (2018) Michael Eagle, Albert Corbett, John Stamper, and Bruce Mclaren. 2018.  Predicting Individualized Learner Models Across Tutor Lessons. In _Proceedings of the 11th International Conference on Educational Data Mining_ , Vol. 11. Raleigh, NC, USA.
  * El Aouifi et al. (2021) H. El Aouifi, M. El Hajji, Y. Es-Saady, and H. Douzi. 2021.  Predicting Learner’s Performance through Video Sequences Viewing Behavior Analysis Using Educational Data-Mining.  _Education and Information Technologies_ 26, 5 (2021), 5799–5814.  https://doi.org/10.1007/s10639-021-10512-4
  * Gan et al. (2020) W. Gan, Y. Sun, X. Peng, and Y. Sun. 2020.  Modeling Learner’s Dynamic Knowledge Construction Procedure and Cognitive Item Difficulty for Knowledge Tracing.  _Applied Intelligence_ 50, 11 (2020), 3894–3912.  https://doi.org/10.1007/s10489-020-01756-7
  * Geden et al. (2020) Michael Geden, Andrew Emerson, Jonathan Rowe, Roger Azevedo, and James Lester. 2020.  Predictive Student Modeling in Educational Games with Multi-Task Learning.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 34, 01 (April 2020), 654–661.  https://doi.org/10.1609/aaai.v34i01.5406
  * Geng et al. (2023) Jing Geng, Huali Yang, and Shengze Hu. 2023.  Noise-Filtering Enhanced Deep Cognitive Diagnosis Model for Latent Skill Discovering.  _Intelligent Automation & Soft Computing_ 37, 2 (2023), 1311–1324.  https://doi.org/10.32604/iasc.2023.038481
  * Gomez et al. (2020) Manuel J. Gomez, José A. Ruipérez-Valiente, Pedro A. Martinez, and Yoon Jeon Kim. 2020.  Exploring the Affordances of Sequence Mining in Educational Games. In _Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality_. ACM, Salamanca Spain, 648–654.  https://doi.org/10.1145/3434780.3436562
  * Gomez et al. (2021) Manuel J. Gomez, José A. Ruipérez-Valiente, Pedro A. Martínez, and Yoon Jeon Kim. 2021.  Applying Learning Analytics to Detect Sequences of Actions and Common Errors in a Geometry Game.  _Sensors_ 21, 4 (Jan. 2021), 1025.  https://doi.org/10.3390/s21041025
  * Gong et al. (2010) Yue Gong, Joseph E. Beck, and Neil T. Heffernan. 2010.  Comparing Knowledge Tracing and Performance Factor Analysis by Using Multiple Model Fitting Procedures. In _Intelligent Tutoring Systems_ _(Lecture Notes in Computer Science)_ , Vincent Aleven, Judy Kay, and Jack Mostow (Eds.). Springer, Berlin, Heidelberg, 35–44.  https://doi.org/10.1007/978-3-642-13388-6_8
  * González-Brenes et al. (2011) J. González-Brenes, W. Duan, and J. Mostow. 2011.  How to Classify Tutorial Dialogue? Comparing Feature Vectors vs. Sequences. In _EDM 2011 - Proceedings of the 4th International Conference on Educational Data Mining_. 169–178.  https://dblp.org/rec/conf/edm/Gonzalez-BrenesMD11
  * Guo et al. (2023) Yi Guo, Tingnian He, Aibin Li, Zhuoran Li, Yixuan Rong, and Guoqi Liu. 2023.  Deep Knowledge Tracking Method Based on DKVTMN-DTCN Model. In _Computer Science and Education_ _(Communications in Computer and Information Science)_ , Wenxing Hong and Yang Weng (Eds.). Springer Nature, Singapore, 623–636.  https://doi.org/10.1007/978-981-99-2446-2_57
  * Ha et al. (2011) Eun Y. Ha, Jonathan P. Rowe, Bradford W. Mott, and James C. Lester. 2011.  Goal Recognition with Markov Logic Networks for Player-Adaptive Games. In _Proceedings of the Seventh AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ _(AIIDE’11)_. AAAI Press, Stanford, California, USA, 32–39.
  * He et al. (2020) Yu He, Xinying Hu, Zhongtian Xu, and and Guangzhong Sun. 2020.  KT-XL: A Knowledge Tracing Model for Predicting Learning Performance Based on Transformer-XL. In _Proceedings of the ACM Turing Celebration Conference - China_ _(ACM TURC ’20)_. Association for Computing Machinery, New York, NY, USA, 175–179.  https://doi.org/10.1145/3393527.3393557
  * Herold et al. (2013) J. Herold, A. Zundel, and T.F. Stahovich. 2013.  Mining Meaningful Patterns from Students’ Handwritten Coursework. In _In Proceedings of the 6th International Conference on Educational Data Mining, EDM 2013_.
  * Hurtig et al. (2022b) Nathan Hurtig, Joseph Hollingsworth, Sarah Blankenship, Eileen Kraemer, Murali Sitaraman, and Jason O. Hallstrom. 2022b.  Network Visualization and Assessment of Student Reasoning About Conditionals. In _Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1_. ACM, Dublin Ireland, 255–261.  https://doi.org/10.1145/3502718.3524793
  * Hurtig et al. (2022a) Nathan Hurtig, Joseph Hollingsworth, and Olga Scrivner. 2022a.  Visualization of Students’ Solutions as a Sequential Network. In _2022 IEEE Global Engineering Education Conference (EDUCON)_. 1189–1194.  https://doi.org/10.1109/EDUCON52537.2022.9766502
  * Jiang et al. (2022) Xuyang Jiang, Yuanxin Ouyang, Zhuang Liu, Wenge Rong, and Zhang Xiong. 2022.  MAKT: Multichannel Attention Networks Based Knowledge Tracing with Representation Learning. In _2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE)_. 292–299.  https://doi.org/10.1109/TALE54877.2022.00055
  * Juhaňák et al. (2019) L. Juhaňák, J. Zounek, and L. Rohlíková. 2019.  Using Process Mining to Analyze Students’ Quiz-Taking Behavior Patterns in a Learning Management System.  _Computers in Human Behavior_ 92 (2019), 496–506.  https://doi.org/10.1016/j.chb.2017.12.015
  * Kesselbacher and Bollin (2019) M. Kesselbacher and A. Bollin. 2019.  Discriminating Programming Strategies in Scratch: Making the Difference between Novice and Experienced Programmers. In _Proceedings of the 14th Workshop in Primary and Secondary Computing Education_.  https://doi.org/10.1145/3361721.3361727
  * Khalil et al. (2022) Mohammad Khalil, Paul Prinsloo, and Sharon Slade. 2022.  The Use and Application of Learning Theory in Learning Analytics: A Scoping Review.  _Journal of Computing in Higher Education_ (Oct. 2022).  https://doi.org/10.1007/s12528-022-09340-3
  * Kinnebrew and Biswas (2011) J.S. Kinnebrew and G. Biswas. 2011.  Modeling and Measuring Self-Regulated Learning in Teachable Agent Environments.  _Journal of E-Learning and Knowledge Society_ 7, 2 (2011), 19–35.
  * Kinnebrew et al. (2014) J.S. Kinnebrew, D.L.C. Mack, G. Biswas, and C.-K. Chang. 2014.  A Differential Approach for Identifying Important Student Learning Behavior Patterns with Evolving Usage over Time.  _Lecture Notes in Computer Science_ 8643 (2014), 281–292.  https://doi.org/10.1007/978-3-319-13186-3_27
  * Kirschner and van Merrienboer (2008) Paul Kirschner and Jeroen J. G. van Merrienboer. 2008.  _Ten Steps to Complex Learning: A New Approach to Instruction and Instructional Design_.  SAGE Publications, Inc., 2455 Teller Road, Thousand Oaks California 91320 United States, I–244–I–253.  https://doi.org/10.4135/9781412964012.n26
  * Knight et al. (2017) Simon Knight, Alyssa Friend Wise, and Bodong Chen. 2017.  Time for Change: Why Learning Analytics Needs Temporal Analysis.  _Journal of Learning Analytics_ 4, 3 (Dec. 2017).  https://doi.org/10.18608/jla.2017.43.2
  * Lajoie et al. (2015) S.P. Lajoie, E.G. Poitras, T. Doleck, and A. Jarrell. 2015.  Modeling Metacognitive Activities in Medical Problem-Solving with Bio World.  _Intelligent Systems Reference Library_ 76 (2015), 323–343.  https://doi.org/10.1007/978-3-319-11062-2_13
  * Latypova (2022) V. Latypova. 2022.  Work with Free Response Implementation Process Analysis Based on Sequential Pattern Mining in Engineering Education. In _2022 VI International Conference on Information Technologies in Engineering Education (Inforino)_. 1–4.  https://doi.org/10.1109/Inforino53888.2022.9782969
  * Li et al. (2022) Shan Li, Juan Zheng, and Susanne Lajoie. 2022.  Temporal Structures and Sequential Patterns of Self-Regulated Learning Behaviors in Problem Solving with an Intelligent Tutoring System.  _Educational Technology & Society_ 25 (March 2022), 1–14.  https://eric.ed.gov/?id=EJ1366535
  * Lin, J. et al. (2022) Lin, J., Singh, S., Sha, L., Tan, W., Lang, D., Gasevic, D., and Chen, G. 2022.  Is it a good move? Mining effective tutoring strategies from human-human tutorial dialogues.  _Future Generation Computer Systems_ 127 (2022), 194–207.  https://doi.org/10.1016/j.future.2021.09.001
  * Liu et al. (2018) M.-C. Liu, C.-H. Yu, J. Wu, A.-C. Liu, and H.-M. Chen. 2018.  Applying Learning Analytics to Deconstruct User Engagement by Using Log Data of MOOCs.  _Journal of Information Science and Engineering_ 34, 5 (2018), 1175–1186.  https://doi.org/10.6688/JISE.201809_34(5).0004
  * Liu et al. (2023) Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, Boyu Gao, Weiqi Luo, and Jian Weng. 2023.  Enhancing Deep Knowledge Tracing with Auxiliary Tasks. In _Proceedings of the ACM Web Conference 2023_ _(WWW ’23)_. Association for Computing Machinery, New York, NY, USA, 4178–4187.  https://doi.org/10.1145/3543507.3583866
  * Lu et al. (2023) Yu Lu, Deliang Wang, Penghe Chen, Qinggang Meng, and Shengquan Yu. 2023.  Interpreting Deep Learning Models for Knowledge Tracing.  _International Journal of Artificial Intelligence in Education_ 33, 3 (Sept. 2023), 519–542.  https://doi.org/10.1007/s40593-022-00297-z
  * Lundgren (2022) Erik Lundgren. 2022.  Latent Program Modeling: Inferring Latent Problem-Solving Strategies from a PISA Problem- Solving Task.  _Journal of Educational Data Mining_ 14, 1 (June 2022), 46–80.  https://doi.org/10.5281/zenodo.6686443
  * Mandlazi et al. (2021) Joel Mandlazi, Ashwini Jadhav, and Ritesh Ajoodha. 2021.  Evaluating Deep Sequential Knowledge Tracing Models for Predicting Student Performance. In _2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE)_. 1–6.  https://doi.org/10.1109/CSDE53843.2021.9718405
  * Matcha et al. (2019) Wannisa Matcha, Dragan Gašević, Nora’ayu Ahmad Uzir, Jelena Jovanović, Abelardo Pardo, Jorge Maldonado-Mahauad, and Mar Pérez-Sanagustín. 2019.  Detection of Learning Strategies: A Comparison of Process, Sequence and Network Analytic Approaches. In _Transforming Learning with Meaningful Technologies_ _(Lecture Notes in Computer Science)_ , Maren Scheffel, Julien Broisin, Viktoria Pammer-Schindler, Andri Ioannou, and Jan Schneider (Eds.). Springer International Publishing, Cham, 525–540.  https://doi.org/10.1007/978-3-030-29736-7_39
  * Meliana and Nurjanah (2018) Selly Meliana and Dade Nurjanah. 2018.  Adopting Good-Learners’ Paths in an Intelligent Tutoring System. In _2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE)_. 877–882.  https://doi.org/10.1109/TALE.2018.8615176
  * Molenaar and Wise (2022) Inge Molenaar and Alyssa Friend Wise. 2022.  Temporal Aspects of Learning Analytics - Grounding Analyses in Concepts of Time.  In _The Handbook of Learning Analytics_ (2 ed.). SoLAR, Vancouver, Canada, 66–76.
  * Morshed Fahid et al. (2021) Fahmid Morshed Fahid, Xiaoyi Tian, Andrew Emerson, Joseph B. Wiggins, Dolly Bounajim, Andy Smith, Eric Wiebe, Bradford Mott, Kristy Elizabeth Boyer, and James Lester. 2021.  Progression Trajectory-Based Student Modeling for Novice Block-Based Programming.  In _Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization_. Association for Computing Machinery, New York, NY, USA, 189–200.  https://doi.org/10.1145/3450613.3456833
  * Motz et al. (2023) Benjamin A. Motz, Yoav Bergner, Christopher A. Brooks, Anna Gladden, Geraldine Gray, Charles Lang, Warren Li, Fernando Marmolejo-Ramos, and Joshua D. Quick. 2023.  A LAK of Direction: Misalignment Between the Goals of Learning Analytics and Its Research Scholarship.  _Journal of Learning Analytics_ (March 2023), 1–13.  https://doi.org/10.18608/jla.2023.7913
  * Mubarak et al. (2021) Ahmed Ali Mubarak, Han Cao, and Salah A.M. Ahmed. 2021.  Predictive Learning Analytics Using Deep Learning Model in MOOCs’ Courses Videos.  _Education and Information Technologies_ 26, 1 (Jan. 2021), 371–392.  https://doi.org/10.1007/s10639-020-10273-6
  * Myers (2021) M.H. Myers. 2021.  Automatic Detection of a Student’s Affective States for Intelligent Teaching Systems.  _Brain Sciences_ 11, 3 (2021), 1–15.  https://doi.org/10.3390/brainsci11030331
  * Nadimpalli et al. (2023) Vamsi Krishna Nadimpalli, Florian Hauser, Dominik Bittner, Lisa Grabinger, Susanne Staufer, and Jürgen Mottok. 2023.  Systematic Literature Review for the Use of AI Based Techniques in Adaptive Learning Management Systems. In _Proceedings of the 5th European Conference on Software Engineering Education_. ACM, Seeon/Bavaria Germany, 83–92.  https://doi.org/10.1145/3593663.3593681
  * Nesbit et al. (2007) John Nesbit, Mingming Zhou, Yabo Xu, and Philip Winne. 2007.  Advancing Log Analysis of Student Interactions with Cognitive Tools.  In _2th Biennial Conference of the European Association for Research on Learning and Instruction (EARLI)_. 2–20.
  * Neyem et al. (2017) Andres Neyem, Juan Diaz-Mosquera, Jorge Munoz-Gama, and Jaime Navon. 2017.  Understanding Student Interactions in Capstone Courses to Improve Learning Experiences. In _Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education_. ACM, Seattle Washington USA, 423–428.  https://doi.org/10.1145/3017680.3017716
  * Ni et al. (2023) Qin Ni, Tingjiang Wei, Jiabao Zhao, Liang He, and Chanjin Zheng. 2023.  HHSKT: A Learner–Question Interactions Based Heterogeneous Graph Neural Network Model for Knowledge Tracing.  _Expert Systems with Applications_ 215 (April 2023), 119334.  https://doi.org/10.1016/j.eswa.2022.119334
  * Nokes et al. (2010) Timothy J. Nokes, Christian D. Schunn, and Micki T. H. Chi. 2010.  Problem Solving and Human Expertise.  In _International Encyclopedia of Education (Third Edition)_ , Penelope Peterson, Eva Baker, and Barry McGaw (Eds.). Elsevier, Oxford, 265–272.  https://doi.org/10.1016/B978-0-08-044894-7.00486-3
  * Norm Lien et al. (2020) Y.-C. Norm Lien, W.-J. Wu, and Y.-L. Lu. 2020.  How Well Do Teachers Predict Students’ Actions in Solving an Ill-Defined Problem in STEM Education: A Solution Using Sequential Pattern Mining.  _IEEE access : practical innovations, open solutions_ 8 (2020), 134976–134986.  https://doi.org/10.1109/ACCESS.2020.3010168
  * Omae et al. (2018) Y. Omae, T. Furuya, K. Mizukoshi, T. Oshima, N. Sakakibara, Y. Mizuochi, K. Yatsushiro, and H. Takahashi. 2018.  Data Mining for Discovering Effective Time-Series Transition of Learning Strategies on Mutual Viewing-Based Learning.  _Journal of Advanced Computational Intelligence and Intelligent Informatics_ 22, 7 (2018), 1046–1055.  https://doi.org/10.20965/jaciii.2018.p1046
  * Owen et al. (2016) E.V. Owen, G. Anton, and R. Baker. 2016.  Modeling User Exploration and Boundary Testing in Digital Learning Games. In _UMAP 2016 - Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization_. 301–302.  https://doi.org/10.1145/2930238.2930271
  * Padrón-Rivera and Rebolledo-Mendez (2015) Gustavo Padrón-Rivera and Genaro Rebolledo-Mendez. 2015.  Identifying Affective Trajectories in Relation to Learning Gains During the Interaction with a Tutoring System. In _Artificial Intelligence in Education_ _(Lecture Notes in Computer Science)_ , Cristina Conati, Neil Heffernan, Antonija Mitrovic, and M. Felisa Verdejo (Eds.). Springer International Publishing, Cham, 756–759.  https://doi.org/10.1007/978-3-319-19773-9_109
  * Page et al. (2021) Matthew J. Page, Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, Jennifer M. Tetzlaff, Elie A. Akl, Sue E. Brennan, Roger Chou, Julie Glanville, Jeremy M. Grimshaw, Asbjørn Hróbjartsson, Manoj M. Lalu, Tianjing Li, Elizabeth W. Loder, Evan Mayo-Wilson, Steve McDonald, Luke A. McGuinness, Lesley A. Stewart, James Thomas, Andrea C. Tricco, Vivian A. Welch, Penny Whiting, and David Moher. 2021.  The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews.  _Systematic Reviews_ 10, 1 (March 2021), 89.  https://doi.org/10.1186/s13643-021-01626-4
  * Qu et al. (2019) S. Qu, K. Li, B. Wu, S. Zhang, and Y. Wang. 2019.  Predicting Student Achievement Based on Temporal Learning Behavior in MOOCs.  _Applied Sciences (Switzerland)_ 9, 24 (2019).  https://doi.org/10.3390/app9245539
  * Rajendran et al. (2018) R. Rajendran, A. Munshi, M. Emara, and G. Biswas. 2018.  A Temporal Model of Learner Behaviors in OELEs Using Process Mining. In _ICCE 2018 - 26th International Conference on Computers in Education, Main Conference Proceedings_. 276–285.  https://apsce.net/icce/icce2018/wp-content/uploads/2018/12/C3-07.pdf
  * Rhienmora et al. (2010) P. Rhienmora, P. Haddawy, P. Khanal, S. Suebnukarn, and M. N. Dailey. 2010.  A Virtual Reality Simulator for Teaching and Evaluating Dental Procedures.  _Methods of Information in Medicine_ 49, 4 (2010), 396–405.  https://doi.org/10.3414/ME9310
  * Rienties et al. (2017) Bart Rienties, Simon Cross, and Zdenek Zdrahal. 2017.  Implementing a Learning Analytics Intervention and Evaluation Framework: What Works?  In _Big Data and Learning Analytics in Higher Education: Current Theory and Practice_ , Ben Kei Daniel (Ed.). Springer International Publishing, Cham, 147–166.  https://doi.org/10.1007/978-3-319-06520-5_10
  * Riofrío-Luzcando and Ramírez (2016) Diego Riofrío-Luzcando and Jaime Ramírez. 2016.  Student Action Prediction for Automatic Tutoring for Procedural Training in 3D Virtual Environments. In _E-Learning, E-Education, and Online Training_ _(Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering)_ , Giovanni Vincenti, Alberto Bucciero, and Carlos Vaz de Carvalho (Eds.). Springer International Publishing, Cham, 200–207.  https://doi.org/10.1007/978-3-319-28883-3_25
  * Riofrío-Luzcando et al. (2017) D. Riofrío-Luzcando, J. Ramírez, and M. Berrocal-Lobo. 2017.  Predicting Student Actions in a Procedural Training Environment.  _IEEE Transactions on Learning Technologies_ 10, 4 (2017), 463–474.  https://doi.org/10.1109/TLT.2017.2658569
  * Sahebi et al. (2014) S. Sahebi, Y. Huang, and P. Brusilovsky. 2014.  Predicting Student Performance in Solving Parameterized Exercises.  _Lecture Notes in Computer Science_ 8474 LNCS (2014), 496–503.  https://doi.org/10.1007/978-3-319-07221-0_62
  * Sahebi et al. (2016) S. Sahebi, Y.-R. Lin, and P. Brusilovsky. 2016.  Tensor Factorization for Student Modeling and Performance Prediction in Unstructured Domain. In _Proceedings of the 9th International Conference on Educational Data Mining, EDM 2016_. 502–506.
  * Saint et al. (2020) John Saint, Dragan Gašević, Wannisa Matcha, Nora’Ayu Ahmad Uzir, and Abelardo Pardo. 2020.  Combining Analytic Methods to Unlock Sequential and Temporal Patterns of Self-Regulated Learning. In _Proceedings of the Tenth International Conference on Learning Analytics & Knowledge_. ACM, Frankfurt Germany, 402–411.  https://doi.org/10.1145/3375462.3375487
  * Sandelowski and Barroso (2007) Margarete Sandelowski and Julie Barroso. 2007.  _Handbook for Synthesizing Qualitative Research_.  Springer Publishing Company, New York, NY.
  * Schatten and Schmidt-Thieme (2016) Carlotta Schatten and Lars Schmidt-Thieme. 2016.  Student Progress Modeling with Skills Deficiency Aware Kalman Filters:. In _Proceedings of the 8th International Conference on Computer Supported Education_. SCITEPRESS - Science and and Technology Publications, Rome, Italy, 31–42.  https://doi.org/10.5220/0005737200310042
  * Schmucker and Mitchell (2022) Robin Schmucker and Tom M Mitchell. 2022.  Transferable Student Performance Modeling for Intelligent Tutoring Systems. In _Proceedings of the 30th International Conference on Computers in Education. Asia-Pacific Society for Computers in Education_. The Asia-Pacific Society for Computers in Education, Kuala Lumpur, Malaysia.
  * Sedrakyan et al. (2020) Gayane Sedrakyan, Jonna Malmberg, Katrien Verbert, Sanna Järvelä, and Paul A. Kirschner. 2020.  Linking Learning Behavior Analytics and Learning Science Concepts: Designing a Learning Analytics Dashboard for Feedback to Support Learning Regulation.  _Computers in Human Behavior_ 107 (June 2020), 105512.  https://doi.org/10.1016/j.chb.2018.05.004
  * Smith et al. (2015) Andy Smith, Wookhee Min, Bradford W. Mott, and James C. Lester. 2015.  Diagrammatic Student Models: Modeling Student Drawing Performance with Deep Learning. In _User Modeling, Adaptation and Personalization_ , Francesco Ricci, Kalina Bontcheva, Owen Conlan, and Séamus Lawless (Eds.). Springer International Publishing, Cham, 216–227.  https://doi.org/10.1007/978-3-319-20267-9_18
  * Song et al. (2022) Xiangyu Song, Jianxin Li, Taotao Cai, Shuiqiao Yang, Tingting Yang, and Chengfei Liu. 2022.  A Survey on Deep Learning Based Knowledge Tracing.  _Knowledge-Based Systems_ 258 (Dec. 2022), 110036.  https://doi.org/10.1016/j.knosys.2022.110036
  * Sun et al. (2019) J.C.-Y. Sun, S.-J. Yu, and C.-H. Chao. 2019.  Effects of Intelligent Feedback on Online Learners’ Engagement and Cognitive Load: The Case of Research Ethics Education.  _Educational Psychology_ 39, 10 (2019), 1293–1310.  https://doi.org/10.1080/01443410.2018.1527291
  * Supianto and Hafis (2019) A.A. Supianto and M. Hafis. 2019.  GTRAS: Graphical Tracking Activity System for Problem-Posing Learning Process Insights. In _2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018_. IEEE, 231–235.  https://doi.org/10.1109/ICACSIS.2018.8618216
  * Tang et al. (2021b) Xueying Tang, Zhi Wang, Jingchen Liu, and Zhiliang Ying. 2021b.  An Exploratory Analysis of the Latent Structure of Process Data via Action Sequence Autoencoders.  _Brit. J. Math. Statist. Psych._ 74, 1 (2021), 1–33.  https://doi.org/10.1111/bmsp.12203
  * Tang et al. (2021a) Yun Tang, Zhengfan Li, Guoyi Wang, and Xiangen Hu. 2021a.  Modeling Learning Behaviors and Predicting Performance in an Intelligent Tutoring System: A Two-Layer Hidden Markov Modeling Approach.  _Interactive Learning Environments_ 0, 0 (Dec. 2021), 1–13.  https://doi.org/10.1080/10494820.2021.2010100
  * Taub and Azevedo (2019) Michelle Taub and Roger Azevedo. 2019.  How Does Prior Knowledge Influence Eye Fixations and Sequences of Cognitive and Metacognitive SRL Processes during Learning with an Intelligent Tutoring System?  _International Journal of Artificial Intelligence in Education_ 29, 1 (March 2019), 1–28.  https://doi.org/10.1007/s40593-018-0165-4
  * Tuti et al. (2019) Timothy Tuti, Chris Paton, Mike English, and Niall Winters. 2019.  Building a Learner Model for a Smartphone-Based Clinical Training Intervention in a Low-Income Context: A Pilot Study. In _Transforming Learning with Meaningful Technologies_ _(Lecture Notes in Computer Science)_ , Maren Scheffel, Julien Broisin, Viktoria Pammer-Schindler, Andri Ioannou, and Jan Schneider (Eds.). Springer International Publishing, Cham, 55–68.  https://doi.org/10.1007/978-3-030-29736-7_5
  * Valle Torre et al. (2023) Manuel Valle Torre, Marcus Specht, and Catharine Oertel. 2023.  Coded Dataset of a Literature Review on Learning Sequence Analysis.  https://doi.org/10.4121/10FC3A10-37AA-4292-818D-468B940B2BB7
  * Valle Torre et al. (2020) Manuel Valle Torre, Esther Tan, and Claudia Hauff. 2020.  edX Log Data Analysis Made Easy: Introducing ELAT: An Open-Source, Privacy-Aware and Browser-Based edX Log Data Analysis Tool. In _Proceedings of the Tenth International Conference on Learning Analytics & Knowledge_ _(LAK ’20)_. Association for Computing Machinery, New York, NY, USA, 502–511.  https://doi.org/10.1145/3375462.3375510
  * Venant et al. (2017) R. Venant, K. Sharma, P. Vidal, P. Dillenbourg, and J. Broisin. 2017.  Using Sequential Pattern Mining to Explore Learners’ Behaviors and Evaluate Their Correlation with Performance in Inquiry-Based Learning.  _Lecture Notes in Computer Science_ 10474 LNCS (2017), 286–299.  https://doi.org/10.1007/978-3-319-66610-5_21
  * Vie and Kashima (2019) Jill-Jenn Vie and Hisashi Kashima. 2019.  Knowledge Tracing Machines: Factorization Machines for Knowledge Tracing.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 33, 01 (July 2019), 750–757.  https://doi.org/10.1609/aaai.v33i01.3301750
  * Wang et al. (2023) Karen D. Wang, Jade Maï Cock, Tanja Käser, and Engin Bumbacher. 2023.  A Systematic Review of Empirical Studies Using Log Data from Open-Ended Learning Environments to Measure Science and Engineering Practices.  _British Journal of Educational Technology_ 54, 1 (2023), 192–221.  https://doi.org/10.1111/bjet.13289
  * Wang et al. (2019) Yu Wang, Tong Li, Congkai Geng, and Yihan Wang. 2019.  Evaluating Student Learning Effect Based on Process Mining. In _Applied Informatics_ _(Communications in Computer and Information Science)_ , Hector Florez, Marcelo Leon, Jose Maria Diaz-Nafria, and Simone Belli (Eds.). Springer International Publishing, Cham, 59–72.  https://doi.org/10.1007/978-3-030-32475-9_5
  * Winne (2010) Philip H. Winne. 2010.  Improving Measurements of Self-Regulated Learning.  _Educational Psychologist_ 45, 4 (Oct. 2010), 267–276.  https://doi.org/10.1080/00461520.2010.517150
  * Wise and Shaffer (2015) Alyssa Friend Wise and David Williamson Shaffer. 2015.  Why Theory Matters More than Ever in the Age of Big Data.  _Journal of Learning Analytics_ 2, 2 (Dec. 2015), 5–13.  https://doi.org/10.18608/jla.2015.22.2
  * Wong et al. (2019) Jacqueline Wong, Mohammad Khalil, Martine Baars, Björn B. de Koning, and Fred Paas. 2019.  Exploring Sequences of Learner Activities in Relation to Self-Regulated Learning in a Massive Open Online Course.  _Computers & Education_ 140 (Oct. 2019).  https://doi.org/10.1016/j.compedu.2019.103595
  * Wu and Ling (2023) Tangjie Wu and Qiang Ling. 2023.  Fusing Hybrid Attentive Network with Self-Supervised Dual-Channel Heterogeneous Graph for Knowledge Tracing.  _Expert Systems with Applications_ 225 (Sept. 2023), 120212.  https://doi.org/10.1016/j.eswa.2023.120212
  * Xia et al. (2023) Zhenchang Xia, Nan Dong, Jia Wu, and Chuanguo Ma. 2023.  Multi-Variate Knowledge Tracking Based on Graph Neural Network in ASSISTments.  _IEEE Transactions on Learning Technologies_ (2023), 1–12.  https://doi.org/10.1109/TLT.2023.3301011
  * Xiao et al. (2022) Wen Xiao, Ping Ji, and Juan Hu. 2022.  A Survey on Educational Data Mining Methods Used for Predicting Students’ Performance.  _Engineering Reports_ 4, 5 (2022), e12482.  https://doi.org/10.1002/eng2.12482
  * Xiong et al. (2016) X. Xiong, S. Zhao, E.G. Van Inwegen, and J.E. Beck. 2016.  Going Deeper with Deep Knowledge Tracing _(Proceedings of the 9th International Conference on Educational Data Mining, EDM 2016)_. 545–550.
  * Yang et al. (2022) Zhenyuan Yang, Shimeng Xu, Changbo Wang, and Gaoqi He. 2022.  Skill-Oriented Hierarchical Structure for Deep Knowledge Tracing. In _2022 IEEE 34th International Conference on Tools with Artificial Intelligence (ICTAI)_. 425–432.  https://doi.org/10.1109/ICTAI56018.2022.00069
  * Zhang et al. (2023) Xiankun Zhang, Yuhu Shang, Yimeng Ren, and Kun Liang. 2023.  Dynamic Multi-Objective Sequence-Wise Recommendation Framework via Deep Reinforcement Learning.  _Complex & Intelligent Systems_ 9, 2 (April 2023), 1891–1911.  https://doi.org/10.1007/s40747-022-00871-x
  * Zhou et al. (2022) Guojing Zhou, Tetsumichi Umada, and Sidney D’Mello. 2022.  What Do Students’ Interactions with Online Lecture Videos Reveal about Their Learning?. In _Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization_ _(UMAP ’22)_. Association for Computing Machinery, New York, NY, USA, 295–305.  https://doi.org/10.1145/3503252.3531315