# Learning Outcomes, Assessment, and Evaluation
in Educational Recommender Systems:
A Systematic Review

Nursultan Askarbekuly1,2, Ivan Luković2

1 Faculty of Computer Science and Engineering, Innopolis University,
Innopolis, Russia, e-mail<EMAIL_ADDRESS>
2 Faculty of Organizational Sciences, The University of Belgrade, Belgrade,
Serbia, e-mails<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we analyse how learning is measured and optimized in
Educational Recommender Systems (ERS). In particular, we examine the target
metrics and evaluation methods used in the existing ERS research, with a
particular focus on the pedagogical effect of recommendations. While
conducting this systematic literature review (SLR), we identified 1395
potentially relevant papers, then filtered them through the inclusion and
exclusion criteria, and finally selected and analyzed 28 relevant papers.
Rating-based relevance is the most popular target metric, while less than a
half of papers optimize learning-based metrics. Only a third of the papers
used outcome-based assessment to measure the pedagogical effect of
recommendations, mostly within a formal university course. This indicates a
gap in ERS research with respect to assessing the pedagogical effect of
recommendations at scale and in informal education settings.

###### Index Terms:

recommendation systems, e-learning, outcome-based, learning outcomes,
assessment, evaluation, systematic literature review.

##  I Introduction

Recommender systems (RS) have been a subject of extensive research and
development for the past two decades. Initially RS gained prominence in
entertainment and e-commerce, and then have also been widely used in the
education [1, 2, 3].

Entertainment, e-commerce, and education have their distinct use cases and
goals, and it is reflected in how the recommendations are made and what
metrics are optimized within each domain. In entertainment, RS were initially
aimed at improving the user ratings [4], and then the focus shifted to user
engagement and retention [5, 6]. In e-commerce, recommendations are made to
increase sales and probability of having return customers [7]. While in
education, recommender systems mainly aim to increase learning through
personalizing educational resources [8].

Like any other software, an educational recommender system (ERS) requires an
appropriate evaluation method aligned with its business goals [9]. Yet
learning is harder to measure than revenue or engagement, as ERS address a
more complex use case, where the recommended item itself does not represent
the end-goal. Instead, a learner’s engagement with a recommended activity
should promote learning [3].

There is no consensus on what is the best way to assess learning from the
pedagogical point of view [10], however one prominent approach to define
learning is through the achievement of the intended learning outcomes [11,
12]. Adopting this outcome-based approach, we believe the evaluation of
educational recommender systems should be rooted in outcome-based assessment
[13].

This systematic literature review (SLR) explores the existing research on
educational recommended systems (ERS). The _objective_ of this SLR is to
investigate _how existing ERS increase learning_. Thus, we review what target
metrics are optimized in existing research on ERS, what evaluation methods
they use, and how the pedagogical effect of recommendations is measured from
the outcome-based point of view.

The _purpose_ behind the SLR is to build a scalable outcome-based educational
recommender system for the informal e-learning domain. This system’s goal is
to automate the process of personalizing and evaluating educational
trajectories within an informal learning domain. This includes the choice of
learning outcomes for a particular learner, recommendation of appropriate
learning activities, and evaluation of the effectiveness of recommendations
through outcome-based assessment. Such automation can be of value to educators
and developers of educational software, as it tackles the non-trivial task of
automatically determining the best educational trajectory in informal domain
at scale.

Section II provides the necessary background and mentions related work.
Section III describes our review methodology. In Section IV, we report the
results, and then discuss them in Section V. Lastly, we conclude the article
in Section VI. For a fast reading of this paper, please see subsections III-A
and III-C for research questions and analysis criteria, then subsection IV-B
for the key results.

##  II Background and Related Work

In this section, we set out to give a reader an overview of all necessary
background knowledge on the topic of our review. We describe the general
theory behind recommender systems, then analyze the previous systematic
reviews on education recommenders, and identify the gap that this work
attempts to fill. We conclude with the subsection on otucome based assessment.

###  II-A Recommender Systems

Recommender systems (RS) are a significant subset of information filtering
systems. They attempt to recommend relevant items by predicting user
preferences or ratings. Traditionally, RS can be categorized into
collaborative filtering (CF), content-based filtering (CBF), knowledge-based
filtering (KB) and their hybridizations [14]. In addition to that, some modern
recommender systems use approaches based on Machine Learning (ML).

Collaborative filtering (CF) methods are based on collecting and analyzing
users’ interaction behaviors with the items, and predicting what users will
like based on how the similarities in ratings between users. Generally, there
are two types of collaborative filtering: user-based (UBCF) and item-based
(IBCF). UBCF algorithms recommend items by finding users with ratings similar
to the target user, and then recommend items that those users have rated
highly. For example, in an educational app, if two students have an
intersection of activities on which they performed well, and one of them
performs well on a new activity, the system might recommend it to the other
student. On the contrary, IBCF models focus on calculating the item-item
similarities through calculating the intersection of user ratings for the
items. For example, in an educational app, if activities A,B, and C have been
highly rated by the same set of users, and a new user likes item A, then the
system might recommend items B and C. Since all recommendations are based on
the user ratings, the advantage of CF is that they do not require any item
metadata or knowledge about the content. However, such systems suffer from the
cold-start problem, in cases where there are no available ratings for new
items or new users.

Content-based filtering (CBF) methods use specific information about each item
to provide recommendations. These could be information describing the content
of the items, like the genre, the director, or the actor of a movie. Similarly
to IBCF, the system recommends items similar to those the user preferred in
the past, however the key difference is that the similarity is calculated
based on the metadata instead of the user ratings. In an educational app, if a
student shows interest in an activity marked with ’Mathematics’ tag, the
system will recommend more content related to ’Mathematics’. The main
difficulty of such systems is that they require the metadata, which is often
difficult or expensive to generate. CBF systems can also suffer from over-
specialization, where they only recommend items similar to those the user has
already rated.

Knowledge-based filtering (KB) systems usually contain some knowledge of the
user needs and how items relate to such needs. They allow to specify
constraints and nuanced relations between items and users, which can be useful
in certain fields. In a way, KB is similar to expert systems [15] in that it
recommends items based on specific rules and knowledge of the domain. An
example of KB recommender is that if a user on an educational app wants to
”prepare for a data science job”, the system might know that certain items and
subjects are a prerequisite to the job, and then recommend the items also
taking the user background knowledge into account. A disadvantage of KB is
that it often requires explicit user inputs and specification of relations
between the modelled entities.

The main difference between the three methods lies in what they consider for
recommendations. CF considers the behavior of users and the interactions
between users and items. It does not need to know anything about the items
themselves. CBF considers descriptions of the items, and relies on past user
behavior and item similarity to make recommendations. Lastly, KBF makes
recommendations based on explicit knowledge about the items, the users, and
the ways these two relate to each other.

Hybrid Systems: These systems combine the aforementioned methods in an attempt
to avoid the limitations. For instance, a CF model might suffer from the ”cold
start” problem, where it’s hard to recommend items to a new user because
there’s not enough data about them yet. CBF or KBF systems can help here by
recommending items based on a few initial preferences set by the new user. CBF
systems can suffer from over-specialization, and CF or KBF methods can help
expand the user’s interests based on ratings or knowledge of relations between
the items and user’s needs.

For a more comprehensive description of various techniques and their
hybridizations, we recommend a foundational survey by Burke [14], who analyzed
existing recommendation techniques in terms of the input data and the
algorithms, and highlighted general rules on when to use each technique, and
examined the range of hybridization methods that have been proposed in the
literature.

Apart from the traditional ones, other more sophisticated approaches to
recommendation systems are also emerging. Context-Aware Recommender Systems
[16] consider user’s additional contextual information like time, location, or
the device used when providing recommendations. This ability to adapt
recommendations based on various contextual factors can potentially enhance
the learning experience and lead to improved learning outcomes [17]. Deep
Learning models [18] can recognize complex patterns by training on large
amounts of data. They are used to create recommender systems that can generate
more thorough metadata by analyzing the content of items or take into account
other factors, which are otherwise ignored by the more traditional systems.
Reinforcement Learning models [19] learn by trial and error. They can be used
to create recommender systems that adapt to changes in user preferences and
behavior over time.

###  II-B Reviews on Recommender Systems in Education

There are several existing reviews of the recommender systems in education,
each of them focusing on an aspect of the process. Drachsler et al [2]
analyzed recommender systems in Technology-Enhanced Learning (TEL) in the
period from year 2000 to 2014, and proposed a classification framework for RS
in TEL. They noted that most RS can be described in terms of the learner
model, domain model, and recommendation algorithm. They also made an important
observation that the background knowledge and learner’s goals should be part
of the learner model. [20] made a concise overview of big data methods and
tools used in recommender systems. Rivera et al [1] conducted a general
systematic mapping and categorized articles according to the educational
context, RS approaches, use case, and evaluation of results. Urdaneta-Ponte et
al [3] also highlight that RS should be tailored to the domain, education
having three of them: formal education, non-formal education, and informal
education. A review by Verbert et al [16] outline the type of contextual
information about the user that can serve as input in the recommendation
process. Thongchotchat et al [21] review how learning styles are used as part
of the learner context in ERS. Da Silva et al [22] review educational
recommender systems in regards to how recommendations have been produced, what
are the most common methods. They touch on evaluation and note the prevalence
of accuracy-based metrics.

However, none of the aforementioned reviews focus on the role of learning
outcomes in the recommendation process, nor discuss the outcome-based
assessment and evaluation in ERS. Furthermore, we found no systematic review
comprehensively examining what is optimized in ERS, i.e. the target metrics
and evaluation methods together with the educational context and supported
tasks.

###  II-C Outcome-based Evaluation in ERS

Any recommender system deployed in a domain should have evaluation metrics
reflecting the unique goals of this system and broader goals of the domain
[14, 9]. Standard recommender systems, such as those used in e-commerce or
entertainment, often prioritize the following metrics:

  * •

Click-Through Rate (CTR): Measures the ratio of users who click on a
recommended item to the number of total users who view the recommendation,
indicating immediate interest but not necessarily long-term engagement or
satisfaction.

  * •

Conversion Rate: In e-commerce, this metric measures the percentage of
recommendations that result in a purchase, focusing on direct revenue impact.

  * •

Engagement and Retention: indicates the system’s success in maintaining users’
interest and facilitating ongoing usage.

  * •

Accuracy Metrics: Such as precision, recall, and F1 score, are common in many
types of recommender systems.

  * •

User Satisfaction and Diversity: reflect content enjoyment or variety from a
broad user-centric perspective.

These standard metrics can be relevant to educational contexts, though their
interpretation may differ. In addition to that, ERS should use more outcome-
based metrics focusing on pedagogical effectiveness [2, 8], such as:

  * •

Learning Gain or Improvement: the increase in knowledge or skills of learners
after engaging with the recommended educational resources, e.g. assessed
through pre-tests and post-tests.

  * •

Concept Mastery and Skill Acquisition: Uses assessments to evaluate how well
learners have mastered specific concepts or acquired particular skills
recommended by the system.

This learning-centered metrics are usually derived from the learning outcomes
(LOs). This outcome-based approach became de-facto European union standard for
learning and teaching [11]. The main idea behind outcome-based learning is
that it is important to ask what competences, knowledge, and skills the
learner gains as the result of the educational process. Biggs [12] also
adheres to this approach, and proposes the idea of constructive alignment
(CA). CA states that both learning activities and assessment should be aligned
with the intended learning outcomes. At, its essence, an intended learning
outcome is a goal that the educator plans for a learner. In our earlier works
[23, 13], we have demonstrated that it is possible to align intended outcomes,
learning activities, and learning assessment when building an educational
product.

To summarize, we assume that educational recommender systems should facilitate
learning, i.e. promote the achievement of learning outcomes. To check this
assumption, we review existing primary ERS research to check what educational
recommenders actually try to impact. By extension, we examine how existing ERS
assess and verify that the intended outcomes are achieved.

##  III Methodology

In this section, we describe the process of our review. Our work uses the SLR
methodology proposed by Kitchenham [24] and partially borrows the paper
structure from Farina et al [25]. We start from stating the research questions
and discuss their relevance to the overall research objective. Then we
describe the analysis criteria based on the research questions. Lastly, we
protocol our search strategy including the databases, search queries, and
inclusion/exclusion criteria used in the review.

###  III-A Research questions

Our research questions follow from the objective and purpose of this SLR
(Section I). _The objective_ of this SLR is to investigate how educational
recommender systems (ERS) model and optimize learning. _The purpose_ behind
the objective is to build an outcome-based ERS for the informal e-learning
domain, including the identification of learning outcomes for a particular
learner, recommendation of learning activities, and evaluation of the
effectiveness of recommendations through outcome-based assessment.

We translated the objective into three research questions keeping in mind the
overall purpose of our investigation:

  * _RQ1: What target metrics are optimized in educational recommender systems (ERS)?_

  * _RQ2: What evaluation methods are used for measuring the effectiveness of ERS?_

  * _RQ3: How do educational recommender systems implement outcome-based assessment?_

_RQ1_ checks the general assumption, embedded in the objective, that ERS
actually aim to optimize learning. It also serves as the basis to further
examine how ERS optimize learning outcomes. _RQ2_ explores how researchers
evaluate the effectiveness of ERS given their target metrics. Lastly, _RQ3_
investigates how outcome-based assessment is implemented in ERS, and how they
relate to the target metric and evaluation methods.

Answering these questions will instruct us on how to measure the effectiveness
of recommendations against learning outcomes as the target metric. This is
coherent with our overall outcome-based approach and the purpose of this SLR
described above.

###  III-B Search strategy

Based on the three research questions we developed a search strategy for
selecting the relevant papers. For our search, we have chosen the following
databases popular in the Computer Science field: IEEE Xplore, ACM Digital
Library, and SpringerLink. We selected IEEE, ACM, and SpringerLink as they are
among the most significant and reputable sources for research in the field of
Computer Science. The research published in these databases is typically of
acceptable quality and peer-reviewed. We did not include other databases such
as DBLP, Scopus, or Web of Science, as there is likely to be an overlap with
the papers indexed in IEEE, ACM, and SpringerLink. We also excluded databases
like Google Scholar and arXiv, as they include a wide range of not peer-
reviewed content, including theses, white papers, and reports, that might not
be as relevant or as rigorously vetted.

Table I reviews the databases used for this SLR in terms of their advantages
and disadvantages. The most important feature of all the three databases is
that they are focused on technology and have sound peer-review process to
guarantee the quality of publications.

TABLE I: Advantages and Disadvantages of Each Database Database |  Advantages |  Disadvantages
---|---|---
IEEE Xplore |  • High-quality, peer-reviewed content. • Specializes in technical literature in electrical engineering, computer science, and electronics. • Advanced search features. |  Most of the content is paid.
SpringerLink |  • Offers access to millions of scientific documents from journals, books, series, protocols, and reference works. • Content from various disciplines. • High-quality, peer-reviewed content. |  Full access usually requires a subscription.
ACM Digital Library |  • Comprehensive collection of ACM’s publications. • Provides access to a broad range of computer science material. • High-quality, peer-reviewed content. |  Full access usually requires a subscription.

Given the topic of our SLR, we have determined the ”educational recommender
systems” to be an appropriate keyphrase.

TABLE II: Keywords and synonyms keyword | synonyms
---|---
educational | learning, training, teaching, tutoring
recommender | intelligent, adaptive
system | -

After a set of trials and preliminary exploration, we also added synonyms to
”educational” and ”recommender”. As the synonyms of ”educational”, we used
”learning”, ”training”, ”teaching”, and ”tutoring”. For ”recommender”, we used
”intelligent” and ”adaptive” (Table II).

Thus, we were able to also cover intelligent tutoring systems and adaptive
learning system, which both recommend and personalize resources and aid for
learners. Using the search queries with boolean operators, we have arrived at
the following search results:

  * •

IEEE Xplore:
(”Document Title”:educat* OR ”Document Title”:learn* OR ”Document
Title”:teach* OR ”Document Title”:train* OR ”Document Title”:tutor*) AND
((”Document Title”:recommend* OR ”Document Title”:intelligent* OR ”Document
Title”:adapt*) NEAR/1 (”Document Title”:system))

  * •

ACM Digital Library:
Title: (educat* OR learn* OR teach* OR train* OR tutor*) AND (recommend* OR
intelligent* OR adapt*) AND (system)

  * •

SpringerLink:
For this database, we had to conduct four separate searches as it only accepts
basic unnested querries
Title: (educat* AND recommend* system)
Title: (learn* AND recommend* system)
Title: (teach* AND recommend* system)
Title: (train* AND recommend* system)
Title: (tutor* AND recommend* system)
Title: (educat* AND intelligent* system)
Title: (learn* AND intelligent* system)
Title: (teach* AND intelligent* system)
Title: (train* AND intelligent* system)
Title: (tutor* AND intelligent* system)
Title: (educat* AND adapt* system)
Title: (learn* AND adapt* system)
Title: (teach* AND adapt* system)
Title: (train* AND adapt* system)
Title: (tutor* AND adapt* system)

To filter the relevant papers we have determined Inclusion and Exlusion
criteria. To be included, a paper must satisfy all of the following Inclusion
Criteria (IC).

  1. 1.

IC1: the work is written in readable English without major understandability
issues AND

  2. 2.

IC2: the work describes a recommender system in the context of education

A work would be excluded if it met any of the following Exclusion Criteria
(EC):

  1. 1.

EC1: the work is similar to a later work by the same authors OR

  2. 2.

EC2: the work is a conference proceeding

  3. 3.

EC3: the work is grey literature (e.g. technical report, dissertation, a
newsletter, a standard).

A potential limitation of our study is the use of only three databases and the
focus on the journal publications, driven by our intention to select the most
relevant and high-quality papers. In the further work, it could be possible to
extend our sources to other databases and conference publications, if we have
research hypotheses and goals justified by our initial approach.

###  III-C Analysis criteria

To answer the research questions, we defined analysis criteria (Table
LABEL:tab:specific_analysis_criteria) to investigate in the selected papers.

TABLE III: Analysis criteria and corresponding research questions ID | Criterion | Research Question
---|---|---
AC1 | Target Metrics | RQ1
AC2 | Evaluation Methods | RQ2
AC3 | Inclusion of Learning Outcomes | RQ3
AC4 | Assessment of Learning Outcomes | RQ3

_AC1_ examines what each of the ERS optimizes and aims to increase. _AC2_
reviews how target metrics are evaluated. _AC1_ and _AC2_ directly correspond
and respectively address _RQ1_ and _RQ2_. Then through _AC3_ and _AC4_ , we
examine whether and how the reviewed systems assess the achievement of
outcomes (_RQ3_). _AC3_ also reviews the role of learning outcomes in the
recommendation process and how they relate to target metrics. While _AC4_ can
help us find techniques for conducting the assessment efficiently. Overall,
these criteria enable us to identify outcome-oriented recommenders and to
explore how they model learning and assessment.

In addition to the criteria specific to our research questions, we also
explored more general ERS attributes such as the context in which the system
is deployed, supported tasks, end-users, and recommendation method used. This
aspects become especially important to examine in outcome-oriented
recommenders.

To summarize the section, we derived research questions from the objective and
purpose of the SLR, then defined search strategy to select relevant papers,
and established the analysis criteria to answer the research questions. In the
following section, we present search results, provide the general overview of
the examined ERS, and then proceed to the main analysis criteria and
corresponding research questions.

##  IV Results

In this section, we present the results of our systematic review. The search
queries defined in Section III-B resulted in the total of 1395 potentially
relevant publications: 678 papers from IEEE Xplore, 34 papers from
SpringerLink, and 683 papers from ACM Digital Library.

Figure 1: Selection flow chart diagram

The initial set of 1395 papers were first filtered through the exclusion
criteria, and then checked against the inclusion criteria to identify the
works most relevant for our SLR (Fig 1). After filtering the papers through
exclusion and inclusion criteria, we identified 28 papers for further analysis
in accordance with criteria defined in Section III-C.

First, we clustered the selected papers based on the publication year and the
scientific database. Fig 2 shows the distribution of the number of papers
published each year. We see that research interest in the area of study has
been steady since 2015, peaking in 2020.

Figure 2: Distribution of papers over years

Fig 3 shows the the distribution of papers over source databases. IEEE has the
highest count, possibly due to having several journals dedicated to
educational technologies, while Springer and ACM have approximately the same
number of relevant publications. We also conducted a subjective quality
evaluation for each of the selected papers, which can be found in Appendix A.
For the reader’s convenience, we also present a descriptive summary of each
paper in Appendix B.

Figure 3: Distribution of the selected papers by database

In the following subsections, we first present the general overview of the
selected papers, and then proceed to the key results related to the research
questions.

###  IV-A General overview: Educational context, supported tasks, and
recommendation methods

We identified three major educational contexts in which the systems were
deployed. For a few papers, we were not able to pinpoint a specific
educational context, as the description was general and applicable to any of
the three context. As a result we had these four context categories.

  * •

Formal Education: Deployed within universities, schools, and online courses,
focusing on curriculum-based learning.

  * •

Informal Learning: Used in non-formal education settings, such as online
forums, social media platforms, or MOOCs, for self-directed learning.

  * •

Lifelong Learning: Recommending resources for continuous professional
development and personal interest learning outside traditional educational
structures.

  * •

Non-specified: Recommenders that did not mention any particular educational
setting.

Figure 4: Educational Context Distribution

Fig 4 shows the educational contexts’ distribution. By far, the most common
context in which the recommender systems were deployed was formal education.
Among them were higher education [26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36,
37, 38, 39, 40, 41], school education [42, 43], children education [44], and
distance learning [40].

There were four papers in the informal learning domain, three of them [45, 46,
47] were a web-based e-learning application, and one social learning platform
[48]. Lastly, three studies did not specify any particular context [49, 50,
51]. In principle, these three can be deployed in any context as long as it is
modelled according to the recommender’s requirements. The only paper from the
lifelong learning domain was about professional training [52].

Figure 5: Recommendation end users

24 out of 28 papers had learners as the end-users receiving recommendations
(Fig. 5). Apart from that, two papers were meant for instructors, one for
researchers, and one for university administration. More specifically, [43]
recommended resources and online communities to school teachers, [35]
suggested course improvement actions to university instructors, [48] suggested
research collaborators based on social profile data, and [31] automated course
allocation and supervisor assignment for university administration.

Figure 6: Supported Tasks Distribution

As for the supported tasks (Fig. 6), the majority of the RS were designed to
recommend educational resources [28, 49, 45, 52, 29, 42, 43, 48, 33, 34, 46,
37, 50, 47, 38, 44, 51, 41]. These resources can encompass a variety of
materials and activities to support learning, from articles and videos to
quizzes and interactive exercises. A similar, but less common task was to
generate personalized exercises [26, 30]. The second most popular task was
providing personalized feedback [27, 53, 29, 42, 37, 40], which is a common
feature for Intelligent Tutoring Systems (ITS). Three systems recommended
courses [36, 32, 39] to help learners navigate their way through a plethora of
available options based on their academic profile and goals. Three more
systems [29, 48, 31] recommended research collaborators and supervisors. One
unique task is the recommendation of course improvement actions to instructors
[35], showing that RS can also be tailored to enhance teaching practices. Four
of the systems mentioned above [29, 42, 48, 37] supported more than one task.

Figure 7: Recommendation Methods: Knowledge-based (KB), Collaborative
filtering (CF), Machine-Learning based (ML), and Hybrid.

Fig. 7 shows recommendation methods used. Knowledge-based (KB) approach is the
most prevalent approach used in 14 papers [26, 27, 53, 45, 52, 29, 42, 48, 31,
33, 47, 44, 40, 41]. By definition, this method requires some insight into the
relations between the recommended items and learners to provide
recommendations. Collaborative Filtering (CF), a method that suggests items
based on user ratings, was used in six papers [28, 43, 32, 34, 46, 38].
Machine Learning-based (ML) recommendation methods, such as classification
algorithms and predictive modeling, were also utilized in six studies [30, 35,
36, 37, 50, 51]. Lastly, two papers employ a hybrid approach that combines
different recommendation strategies [49, 39]. Interestingly, no papers used
Content-based filtering, possibly due to the fact that the knowledge-based and
hybrid approaches serve as a more sophisticated replacement, especially when
using ontologies to model the domain items [39, 52, 47, 40].

To summarize the general overview, the main context is formal education, the
end-user is mainly learner, and most popular supported task is recommending
educational resources. There were also a few papers in the informal learning
domain, while the second most popular supported task was providing
personalized feedback such as hints. Among the recommendation methods,
knowledge-based approach is the most popular, followed by collaborative
filtering and machine learning.

In the next subsection, we review the key analysis results directly related
the research questions.

###  IV-B Key results: Analysing the Research Questions

In this subsection, we review the analysis criteria directly relating to the
research questions (section III-A). The analysis data is shown in Table IV.

TABLE IV: Main results based on the analysis criteria Ref |  Target Metric |  Includes LOs |  Evaluation Method |  Assesses LOs
---|---|---|---|---
[37] |  Academic Performance Self-Evaluation |  Yes |  Formal Assessment Qualitative |  Yes
[41] |  Academic Performance User Rating |  Yes |  Formal Assessment Experimental Qualitative |  Yes
[42] |  Academic Performance User Satisfaction Self-Evaluation |  Yes |  Formal Assessment Experimental Qualitative |  Yes
[34] |  Academic Performance User Satisfaction |  Yes |  Formal Assessment Experimental Qualitative |  Yes
[29] |  Academic Performance |  Yes |  Formal Assessment Experimental |  Yes
[53] |  Probability of Learning |  Yes |  Simulation |  Yes
[44] |  Probability of Learning |  Yes |  Simulation |  Yes
[27] |  Probability of Learning |  Yes |  No evaluation |  Yes
[30] |  Probability of Learning |  Yes |  No evaluation |  Yes
[51] |  Academic Performance |  Yes |  Benchmarking |  No
[35] |  Academic Performance |  Yes |  Experimental |  No
[52] |  Expert rating |  Yes |  Experimental |  No
[49] |  User Rating |  Yes |  Benchmarking |  No
[36] |  Academic Performance |  No |  Experimental |  No
[40] |  Expert rating |  No |  Experimental |  No
[46] |  User Rating User Satisfaction |  No |  Experimental Qualitative |  No
[45] |  User Rating |  No |  Benchmarking |  No
[50] |  User Rating |  No |  Benchmarking |  No
[39] |  User Rating |  No |  Benchmarking |  No
[28] |  User Rating |  No |  Experimental |  No
[31] |  User Rating |  No |  Experimental |  No
[33] |  User Rating |  No |  Experimental |  No
[32] |  User Rating |  No |  No evaluation |  No
[48] |  User Rating |  No |  Qualitative |  No
[47] |  User Rating |  No |  Qualitative |  No
[38] |  User Rating |  No |  Qualitative |  No
[26] |  User Satisfaction |  No |  Qualitative |  No
[43] |  User Satisfaction |  No |  Qualitative |  No

For a more detailed account of results, please see our Github repository with
the CSV file containing all collected review data
111https://github.com/nurlingo/slr-ers/blob/main/data.csv.

_RQ1: What target metrics are optimized in educational recommender systems
(ERS)?_

To answer _RQ1_ , we examined Target Metrics (_AC1_) that each of the examined
recommenders aimed to increase.

Figure 8: AC1: Target Metrics that ERS optimize

The most popular target metric is user-rating. Almost half of the papers used
it when evaluating the effectiveness of their system [28, 49, 45, 48, 31, 32,
33, 46, 50, 47, 38, 39]. The user ratings were collected either explicitly by
allowing users to rate the items or implicitly by logging the desired user
behaviours such as clicks. Two more papers [40, 52] relied on expert opinion
and compared their system’s performance against recommendations given by
experts. Importantly, both user and expert ratings were in fact proxies for
relevance to the user preferences and needs.

A similar user-centered set of target metrics were user-satisfaction [42, 43,
34, 46] and self-evaluation [42, 37]. These are similar to user rating-based
relevance, however present a more diverse set of criteria to evaluate the
recommendations’. The users were asked about the effect on their learning and
their satisfaction. Interestingly, [42] demonstrates that higher satisfaction
and self-evaluation do not necessarily correlate with academic performance.

Twelve papers used more learning-centered metrics. The most popular among them
one was academic performance [29, 42, 34, 35, 36, 37, 51, 41], mostly
implemented within the context of higher education and measured through formal
course assessment.

Another learning-centered metric was probability of learning that assesses how
likely it is that a learner has mastered a concept or a skill based on
observing the learner behaviour. Three papers [27, 53, 30] implemented it in
the context of an intelligent tutoring or adaptive learning system, where they
modelled the learner through knowledge states and observed his performance to
update the state probabilities. While [44] used the number of interactions
with other learners to determine the likelihood of learning.

Worth noting that several papers [42, 34, 46, 37, 41] used two or more target
metrics, thus allowing for a more comprehensive view that can cover both user-
centered and learning-centered perspectives.

_RQ2: What evaluation methods are used for measuring the effectiveness of
ERS?_

To answer RQ2, we reviewed the evaluation methods (_AC2_) in selected papers,
and placed them into a few broad categories.

Figure 9: AC2: Evaluation methods used

A significant portion of the studies [28, 52, 29, 42, 31, 33, 35, 36, 34, 46,
41, 40] have opted for experimental evaluation methods including online
controlled experiments with pre- and post-tests, as well as offline data
splits into training and test datasets.

Ten studies [26, 42, 48, 43, 37, 34, 46, 47, 38, 41] have utilized qualitative
measurements questionnaires to understand the user experience and satisfaction
with ERS.

Five studies [29, 42, 37, 34, 41] used formal academic assessment, mostly in
combination with some experimental setting, whereby learners were split into
groups and their performance was assessed through course work, tests, and
quizzes. Five more studies [49, 45, 50, 39, 51] employed benchmarking
approaches to evaluate their systems against other known algorithms using
public datasets. Two studies [53, 44] used simulation (a virtual model of the
system) and studied its behavior under various conditions. While a few studies
had no relevant evaluation stage [27, 30, 32].

Some researchers used mixed methods to provide a more comprehensive evaluation
of their ERS. For instance, [42, 37, 46, 43, 48] combined experimental methods
and qualitative metrics to bridge the gap between objective measurements and
subjective user experiences.

Figure 10: Evaluation datasets used in the papers

As a side note, we noticed that the majority of studies relied on undisclosed,
internal datasets (Fig. 10) when conducting evaluation. One of the studies
used famous MovieLens [49], which is a dataset from an unrelated domain. This
does not allow for the reproducibility of the studies, and possibly indicates
a lack of available educational datasets.

_RQ3: How do educational recommender systems implement outcome-based
assessment?_

To answer _RQ3_ , we reviewed whether the systems used learning outcomes
(_AC3_) and outcome-based assessment (_AC4_) within the recommendation
process.

Figure 11: Inclusion and Assessment of Learning Outcomes

As shown in Fig. 11, almost half of the papers made use of learning outcomes.
Six of these papers [43, 29, 35, 37, 41, 34] recommended items within a
university course, and therefore reused the intended outcomes readily
available for the course.

Five other papers [51, 42, 27, 53, 30, 44] defined outcomes manually as
concepts or skills relevant to the subject being studied and modelled them as
part of the recommendation process. A similar approach was taken by [52, 49],
who defined outcomes in the form of professional competencies and used them as
the basis for making recommendations. The rest of the papers either did not
include learning outcomes or mode no explicit mention of their use.

As for the outcome-based assessment, roughly a third of papers measured the
achievement of learning outcomes (Fig. 11). Seven of them [49, 29, 37, 41, 34,
42, 53] used formal assessment readily available as part of a university
course or professional training. In that case, the learners were split into
several group for the evaluation through a controlled experiment.

Three more papers could be said to have automated outcome-based assessment.
[27, 30] assessed the solution to automatically generated programming
exercises. In their case, different parts of the expected solution were linked
to a particular learning outcome, and the probability of achieving the outcome
was updated based on observing which parts of the program were written
correctly. While [44] took an indirect approach and approximated the
assessment by putting a threshold on how many activations (engagements with
other learners) should a learner have to be marked as someone who learnt a
particular concept.

Interestingly, four papers included learning outcomes, but did not conduct the
assessment. [51, 35] recommended items that had the highest possibility of
success based on historic data, but did not verify their predictions through
assessment. While [52, 49] prioritized relevance-based target metrics.

To summarize, our objective was to review how educational recommender systems
(ERS) optimize learning. We reviewed the target metrics used in existing ERS
with the corresponding evaluation methods. Since we define learning as the
achievement of learning outcomes, we also reviewed how they implemented
outcome-based assessment. The summary results of our review are presented in
Table IV. In the following section, we elaborate our findings in the light of
research questions, offer interpretations, and discuss some of the limitations
and further work.

##  V Discussion

Our SLR on Educational Recommender Systems (ERS) reveals several important
trends, as well as some potential gaps that merit further exploration. In the
general overview, we found that ERS were mostly deployed in the context of
formal education, the end-users were mostly learners, and the recommended
items were mainly educational resources. The most common approach was
knowledge-based, which shows that researchers try to apply their understanding
of the domain to improve the recommendations. Since these overview results
(Section IV-A) have been covered by and generally agree with previous
systematic reviews on ERS [3, 22], we immediately proceed to discuss the key
results related to the research questions of our SLR.

###  V-A Lack of focus on learning

For the target metric, most of the reviewed ERS optimized relevance of the
served items, then learning, then other user-centric metrics such as
satisfaction. This raises a question: why relevance is the most popular target
metric in the domain of education, and not a metric directly related to
learning? Given that the learner is the most common end-user and educational
resources are the main focus of recommendations, one might assume that
learning should be primary focus, yet only 12 papers out of 28 optimized
learning.

One possible explanation is the historical context within which recommender
systems emerged. The focus on the relevance and classification or prediction
performance metrics aligns with the needs in entertainment and e-commerce,
where the recommended item is closer to the business goal. Possibly, the
researchers transferred the algorithms together with evaluation methods
without necessary adjustments to the domain. However, even in these initial
domains, they have come to use more sophisticated long-term metrics such as
retention and promoting habitual use [54]. This means that educational
recommenders should also adjust the evaluation metrics, i.e. move beyond
relevance and closer to the business goals within the domain. This highlights
the need for determining a comprehensive overall evaluation criteria (OEC)
[55] that includes educational, user-centric, and product-oriented metrics.
Interestingly, we can already see that some of the reviewed papers used a
combination of several target metrics to either compare between them or to
have a more comprehensive view. This also correlates with the recent trend of
having context-rich user profiles in ERS [16], whereby researchers are
exploring new significant user properties to base the recommendations on. As
further work, we can combine the outcome-based approach with context-rich
recommendations to determine the best predictors. This would answer the
question of where should the attention go when recommending the items and
assessing their effectiveness from the pedagogical point of view.

Another possible explanation is that the researchers operate on the assumption
that relevance leads to engagement, while engagement leads to greater learning
outcomes. This is partially supported by previous research [56]. However, this
approach does not allow to differentiate between more and less effective
learning activities, since it assumes the activity to be effective as long as
the learner is engaged. Several of the reviewed papers proposed more holistic
user-centric metrics to ensure the recommendations are having an overall
positive effect. However these are still not outcome-focused, as both
engagement, satisfaction, and positive self-evaluation has been shown to not
necessarily correlate with learning [42]. Moreover, using indirect metrics to
measure learning presents a limitation from the scientific point of view,
since having indirect observations will necessarily be less conclusive.
Evaluating the learning through more direct means would enable us to determine
the most effective types of engagement, context elements, and recommendation
methods from the pedagogical point of view. This also agrees with the
principal idea in engineering that metrics should correspond to business goals
[9], and the similar idea in pedagogy that learning activities and assessment
should be aligned with the intended outcomes [12].

If standard engagement metrics are used, they should be adjusted to focus more
on educational relevance than on general popularity or click-through rate. For
example, engagement metrics can include time spent on learning tasks,
completion rates of recommended courses or activities, and interaction rates
with the system, hence ultimately contributing to and correlating with the
achievement of learning outcomes. Retention can be reformulated as progression
or completion rates to track the learners’ progress in a course or program
over time. Similarly user-centric metrics can be important in ERS, if the
satisfaction is closely tied to educational value and personal learning
progress. Personalization would then mean the alignment of recommendations
with individual learning goals, styles, or other contextual learner
properties. However, none of the examined papers frame the discussion of
metrics in this manner. An important direction for further work is to
investigate the correlation between learning, engagement, and satisfaction,
and thus attemp to identify the types of indirect metrics most correlating
with learning to optimize towards them.

Yet another explanation is that learning is hard to measure [10, 57]. In other
words, despite understanding the importance of assessing the learning and
using it as the principle optimization target, the researchers are bound to
use other means. It can be argued that learning assessment is a wicked problem
[58], which is hard to define and measure definitively. It can also be argued
that the outcome-oriented approach is not always applicable, for example in
affective learning domain [59], where learning outcomes are open-ended rather
than pre-defined by the instructor. However, open-endedness does not mean
absence, and even in the affective learning the best practice is to provide a
way for the learner to self-determine, document, and reflect upon the achieved
learning outcomes.

Being the proponents of the outcome-based approach, we recognize that
designing well-aligned learning assessment is a demanding and resource-
intensive task. It requires domain knowledge, understanding the learner’s
level and needs, determining the intended outcomes, and designing the
assessment activities. This brings us to _RQ2_ and _RQ3_ which explore the
evaluation methods used by ERS and how outcome-based assessment is
implemented.

###  V-B The relation between target metrics and evaluation methods

The reviewed research had several evaluation methods evenly distributed over
the papers. The menthods included experimental, qualitative, simulation,
benchmarking, and mixed approaches. This distribution reflects a broad
consensus on the importance of multifaceted evaluation.

Our main interest was in the methods used with the learning-centered target
metrics. Those were mostly gathered through formal assessment combined with
experimental methods. Formal assessment is possible in formal education, where
the learning outcomes and corresponding assessment activities are defined at
the outset. However, this can be challenging in informal learning domain due
to a lack of structured assessment mechanisms. Two of the reviewed works
attempted to assess the probability of learning through simulation, and two
more works omitted evaluation of their proposed system despite having a
learning-centric target metric. This supports the explanation above that
learning is hard to measure, especially so in the informal domain, as the ERS
that evaluated learning effectiveness mostly did so in the context of academic
courses.

An interesting question is how much the feasibility of an evaluation method
determines the target metric. In the ideal world, the metrics should proceed
from goals, and then the corresponding method should be chosen. Possibly,
there are not many public datasets sufficiently rich to allow a comprehensive
evaluation of recommendations from the learning point of view. Most of the
open educational datasets used in the papers [43, 39, 45, 31] focus on
engagement and ratings data. Therefore, the absence of appropriate datasets
might be a factor affective the choice of target metrics. While having such
datasets would enable researchers to conduct more meaningful evaluations of
educational recommender systems, and promote the essential objective of
education - ensuring the achievement of specific learning outcomes.

###  V-C Three common ways to implement outcome-based assessment

For outcome-based assessment, we can essentially outline three approaches that
were used within the reviewed ERS: outcomes and assessment were already in
place as part of a formal course, or the assessment exercises automatically
generated and graded, or their achievement was concluded from indirect
observations.

The first approach is when ERS is deployed within a structured academic course
or professional training. In this case, both the outcomes and corresponding
formal assessment are already defined, and, in principle, need not to be part
of the recommendation process, rather it is enough to log the assessment
results and optimize the recommendations accordingly. The evaluation then will
be done through splitting the learners into experimental groups. This can work
given that the course organizers correctly define the outcomes, and align the
assessment activities accordingly. However, this approach does not resolve the
question of efficiency for informal settings.

The second approach is essentially a knowledge-based method, where the
outcomes, assessment activities and relation between them are modelled within
the system. This approach necessitates a dynamic learner, where learner
completes the assessment exercises, and the mastery of corresponding outcomes
is concluded based on the results. However, this approach requires knowledge
of the domain and significant manual effort when composing the outcomes,
assessment, and their relations. Therefore, the composition of outcomes and
assessment exercises, mapping between them should ideally be automated. If the
automation is viable, this can be an effective approach even within the
informal learning domain at scale. This is a potential area of further work.

The third approach is to replace traditional assessment with other indirect
measures indicative of learning, such as collaboration with other users.
However, such a replacement has to strongly correlate with learning. Which
raises concerns similar to what has been discussed above in regards to using
relevance as the proxy for learning. Still, if such correlation is
demonstrated in a specific setting, then the indirect measure could possibly
be used instead of assessment activities. Possibly assessment initially be
used to identify the most important user-context elements, to then predict the
learning gains and maximize the effect of recommendation methods from the
pedagogical point of view.

In summary:

  * •

There appear to be a gap in outcome-based evaluation of educational
recommender systems, especially outside of a formal educational setting. The
existing ERS tend to optimize standard recommendation metrics such as rating-
based relevance, click-Through Rate (CTR), and user-satisfaction;

  * •

ERS should use comprehensive evaluation criteria reflecting the system and
domain goals. The main focus should be on learning-oriented metrics, while the
standard recommender metrics can be reinterpreted to align with learning.

  * •

There are three common approaches for outcome-based assessment in ERS, namely
formal academic assessment, knowledge-based modelling, and using indirect
measures. Each of the approaches has its limitations and requirements for the
deployment context. The first two approaches raise the question of efficiency,
while the third approach necessitate the identification of strong predictors
of learning.

Therefore, we propose three significant areas of focus for further work:

  1. 1.

Defining a comprehensive overall evaluation criteria (OEC) that incorporates
outcome-based, user-centric, and product-oriented metrics;

  2. 2.

Identifying the key predictors of learning, i.e. metrics and user-context
properties that strongly correlate with learning when recommending items;

  3. 3.

Experimenting with various efficient ways to implement outcome-based
assessmetns, possibly through automation or correlated metrics.

##  VI Conclusion

The SLR explores how learning is measured and optimized in educational
recommender systems (ERS). In particular, we investigated the target metrics
optimized in ERS, the evaluation methods used for measuring their
effectiveness, and how outcome-based assessment is implemented in ERS. Our
review began with 1395 papers, which were narrowed down to 28 relevant works.
We found rating-based relevance to be the prevalent target metric. We also
found that only a third of papers implemented outcomes-based assessment,
possibly indicating a gap. Further work in ERS should propose comprehensive
evaluation criteria that prioritize learning-related metrics, while still
covering user-centric and product-oriented perspectives by adjusting them to
the domain. Finding efficient ways to conduct outcome-based assessment appears
to be another critical aspect to address the gap.

## References

  * [1] A. C. Rivera, M. Tapia-Leon, and S. Lujan-Mora, “Recommendation systems in education: a systematic mapping study,” in International Conference on Information Technology & Systems, pp. 937–947, Springer, 2018.
  * [2] H. Drachsler, K. Verbert, O. C. Santos, and N. Manouselis, “Panorama of recommender systems to support learning,” in Recommender systems handbook, pp. 421–451, Springer, 2015.
  * [3] M. C. Urdaneta-Ponte, A. Mendez-Zorrilla, and I. Oleagordia-Ruiz, “Recommendation systems for education: systematic review,” Electronics, vol. 10, no. 14, p. 1611, 2021.
  * [4] J. Bennett, S. Lanning, et al., “The netflix prize,” in Proceedings of KDD cup and workshop, vol. 2007, p. 35, New York, 2007.
  * [5] B. P. Knijnenburg, M. C. Willemsen, Z. Gantner, H. Soncu, and C. Newell, “Explaining the user experience of recommender systems,” User modeling and user-adapted interaction, vol. 22, pp. 441–504, 2012.
  * [6] E. Maslowska, E. C. Malthouse, and L. D. Hollebeek, “The role of recommender systems in fostering consumers’ long-term platform engagement,” Journal of Service Management, vol. 33, no. 4/5, pp. 721–732, 2022.
  * [7] B. Xiao and I. Benbasat, “E-commerce product recommendation agents: Use, characteristics, and impact,” MIS quarterly, pp. 137–209, 2007.
  * [8] S. Garcia-Martinez and A. Hamou-Lhadj, “Educational recommender systems: A pedagogical-focused perspective,” in Multimedia Services in Intelligent Environments, pp. 113–124, Springer, 2013.
  * [9] V. R. B. G. Caldiera and H. D. Rombach, “The goal question metric approach,” Encyclopedia of software engineering, pp. 528–532, 1994.
  * [10] B. Kizlik, “Measurement, assessment, and evaluation in education,” Retrieved October, vol. 10, p. 2015, 2012.
  * [11] E. Union, “Europass tools: European qualifications framework.” https://europa.eu/europass/en/europass-tools/european-qualifications-framework.  [Accessed: May 15, 2023].
  * [12] J. Biggs, “What the student does: Teaching for enhanced learning,” Higher education research & development, vol. 31, no. 1, pp. 39–55, 2012.
  * [13] N. Askarbekuly, A. Sadovykh, and M. Mazzara, “Combining two modelling approaches: Gqm and kaos in an open source project,” in Open Source Systems: 16th IFIP WG 2.13 International Conference, OSS 2020, Innopolis, Russia, May 12–14, 2020, Proceedings 16, pp. 106–119, Springer, 2020.
  * [14] R. Burke, “Hybrid recommender systems: Survey and experiments,” User modeling and user-adapted interaction, vol. 12, no. 4, pp. 331–370, 2002.
  * [15] L. Fernandez-Luque, R. Karlsen, and L. K. Vognild, “Challenges and opportunities of using recommender systems for personalized health education,” Medical informatics in a united and healthy Europe, pp. 903–907, 2009.
  * [16] K. Verbert, N. Manouselis, X. Ochoa, M. Wolpers, H. Drachsler, I. Bosnic, and E. Duval, “Context-aware recommender systems for learning: A survey and future challenges,” IEEE Transactions on Learning Technologies, vol. 5, no. 4, pp. 318–335, 2012.
  * [17] S. Raza and C. Ding, “Progress in context-aware recommender systems — an overview,” Computer Science Review, vol. 31, pp. 84–97, 2019.
  * [18] L. Salau, M. Hamada, R. Prasad, M. Hassan, A. Mahendran, and Y. Watanobe, “State-of-the-art survey on deep learning-based recommender systems for e-learning,” Applied Sciences, vol. 12, no. 23, p. 11996, 2022.
  * [19] M. M. Afsar, T. Crump, and B. Far, “Reinforcement learning based recommender systems: A survey,” ACM Computing Surveys, vol. 55, no. 7, pp. 1–38, 2022.
  * [20] S. Dwivedi and V. K. Roshni, “Recommender system for big data in education,” in 2017 5th National Conference on E-Learning & E-Learning Technologies (ELELTECH), pp. 1–4, IEEE, 2017.
  * [21] V. Thongchotchat, Y. Kudo, Y. Okada, and K. Sato, “Educational recommendation system utilizing learning styles: A systematic literature review,” IEEE Access, vol. 11, pp. 8988–8999, 2023.
  * [22] F. L. da Silva, B. K. Slodkowski, K. K. A. da Silva, and S. C. Cazella, “A systematic literature review on educational recommender systems for teaching and learning: research trends, limitations and opportunities,” Education and Information Technologies, vol. 28, no. 3, pp. 3289–3328, 2023.
  * [23] N. Askarbekuly, A. Solovyov, E. Lukyanchikova, D. Pimenov, and M. Mazzara, “Building an educational product: constructive alignment and requirements engineering,” in Advances in Artificial Intelligence, Software and Systems Engineering: Proceedings of the AHFE 2021 Virtual Conferences on Human Factors in Software and Systems Engineering, Artificial Intelligence and Social Computing, and Energy, July 25-29, 2021, USA, pp. 358–365, Springer, 2021.
  * [24] B. Kitchenham, “Procedures for performing systematic reviews,” Keele, UK, Keele University, vol. 33, no. 2004, pp. 1–26, 2004.
  * [25] M. Farina, A. Gorb, A. Kruglov, and G. Succi, “Technologies for gqm-based metrics recommender systems: A systematic literature review,” IEEE Access, 2022.
  * [26] S. Fischer, “Course and exercise sequencing using metadata in adaptive hypermedia learning systems,” vol. 1, p. 5–es, mar 2001.
  * [27] S. M. Parvez and G. D. Blank, “A pedagogical framework to integrate learning style into intelligent tutoring systems,” J. Comput. Sci. Coll., vol. 22, p. 183–189, jan 2007.
  * [28] F. Abel, I. I. Bittencourt, E. Costa, N. Henze, D. Krause, and J. Vassileva, “Recommendations in online discussion forums for e-learning systems,” IEEE Transactions on Learning Technologies, vol. 3, no. 2, pp. 165–176, 2010.
  * [29] N. Hoic-Bozic, M. Holenko Dlab, and V. Mornar, “Recommender system and web 2.0 tools to enhance a blended learning model,” IEEE Transactions on Education, vol. 59, no. 1, pp. 39–44, 2016.
  * [30] D. S. Myers and N. Chatlani, “Implementing an adaptive tutorial system for coding literacy,” J. Comput. Sci. Coll., vol. 33, p. 260–267, dec 2017.
  * [31] H. Samin and T. Azim, “Knowledge based recommender system for academia using machine learning: A case study on higher education landscape of pakistan,” IEEE Access, vol. 7, pp. 67081–67093, 2019.
  * [32] K. Dahdouh, A. Dakkak, L. Oughdir, and A. Ibriz, “Large-scale e-learning recommender system based on spark and hadoop,” Journal of Big Data, vol. 6, no. 1, p. 2, 2019.
  * [33] S. Pariserum Perumal, G. Sannasi, and K. Arputharaj, “An intelligent fuzzy rule-based e-learning recommendation system for dynamic user interests,” The Journal of Supercomputing, vol. 75, no. 8, pp. 5145–5160, 2019.
  * [34] J. Cárdenas-Cobo, A. Puris, P. Novoa-Hernández, J. A. Galindo, and D. Benavides, “Recommender systems and scratch: An integrated approach for enhancing computer programming learning,” IEEE Transactions on Learning Technologies, vol. 13, no. 2, pp. 387–403, 2020.
  * [35] N. Yanes, A. M. Mostafa, M. Ezz, and S. N. Almuayqil, “A machine learning-based recommender system for improving students learning experiences,” IEEE Access, vol. 8, pp. 201218–201235, 2020.
  * [36] A. J. Fernández-García, R. Rodríguez-Echeverría, J. C. Preciado, J. M. C. Manzano, and F. Sánchez-Figueroa, “Creating a recommender system to support higher education students in the subject enrollment decision,” IEEE Access, vol. 8, pp. 189069–189088, 2020.
  * [37] P.-H. Lin and S.-Y. Chen, “Design and evaluation of a deep learning recommendation based augmented reality system for teaching programming and computational thinking,” IEEE Access, vol. 8, pp. 45689–45699, 2020.
  * [38] N. Vedavathi and K. M. Anil Kumar, “An efficient e-learning recommendation system for user preferences using hybrid optimization algorithm,” Soft Computing, vol. 25, no. 14, pp. 9377–9388, 2021.
  * [39] M. Amane, K. Aissaoui, and M. Berrada, “Ersdo: E-learning recommender system based on dynamic ontology,” vol. 27, p. 7549–7561, jul 2022.
  * [40] M. Belver, A. Manjarrés, A. Barbarelli, and S. Pickin, “Requirements elicitation based on psycho-pedagogical theatre for context-sensitive affective educational recommender systems,” IEEE Access, vol. 11, pp. 76284–76299, 2023.
  * [41] F. Okubo, T. Shiino, T. Minematsu, Y. Taniguchi, and A. Shimada, “Adaptive learning support system based on automatic recommendation of personalized review materials,” IEEE Transactions on Learning Technologies, vol. 16, no. 1, pp. 92–105, 2023.
  * [42] Y. Long and V. Aleven, “Educational game and intelligent tutoring system: A classroom study and comparative design analysis,” ACM Trans. Comput.-Hum. Interact., vol. 24, apr 2017.
  * [43] S. Fazeli, H. Drachsler, M. Bitter-Rijpkema, F. Brouns, W. v. d. Vegt, and P. B. Sloep, “User-centric evaluation of recommender systems in social learning platforms: Accuracy is just the tip of the iceberg,” IEEE Transactions on Learning Technologies, vol. 11, no. 3, pp. 294–306, 2018.
  * [44] M. Moradi, K. R. Fard, and M. Y. Akhlaqi, “A recommender system method for children’s education using mobile technology,” IEEE Access, vol. 10, pp. 123679–123696, 2022.
  * [45] K. Niemann and M. Wolpers, “Creating usage context-based object similarities to boost recommender systems in technology enhanced learning,” IEEE Transactions on Learning Technologies, vol. 8, no. 3, pp. 274–285, 2015.
  * [46] M. Bustos López, G. Alor-Hernández, J. L. Sánchez-Cervantes, M. A. Paredes-Valverde, M. d. P. Salas-Zárate, and T. Bickmore, “Edurecomsys: An educational resource recommender system based on collaborative filtering and emotion detection,” Interacting with Computers, vol. 32, no. 1, pp. 407–432, 2020.
  * [47] J. Joy, N. S. Raj, and R. V. G., “Ontology-based e-learning content recommender system for addressing the pure cold-start problem,” vol. 13, no. 3, 2021.
  * [48] C. K. Pereira, F. Campos, V. Ströele, J. M. N. David, and R. Braga, “Broad-rsi – educational recommender system using social networks interactions and linked data,” Journal of Internet Services and Applications, vol. 9, p. 7, 03 2018.
  * [49] D. Wu, J. Lu, and G. Zhang, “A fuzzy tree matching-based personalized e-learning recommender system,” IEEE Transactions on Fuzzy Systems, vol. 23, no. 6, pp. 2412–2426, 2015.
  * [50] W. Intayoad, C. Kamyod, and P. Temdee, “Reinforcement learning based on contextual bandits for personalized online learning recommendation systems,” Wireless Personal Communications, vol. 115, no. 4, pp. 2917–2932, 2020.
  * [51] S. Liu, S. Liu, Z. Yang, J. Sun, X. Shen, Q. Li, R. Zou, and S. Du, “Heterogeneous evolution network embedding with temporal extension for intelligent tutoring systems,” ACM Trans. Inf. Syst., vol. 42, nov 2023.
  * [52] P. Montuschi, F. Lamberti, V. Gatteschi, and C. Demartini, “A semantic recommender system for adaptive learning,” IT Professional, vol. 17, no. 5, pp. 50–58, 2015.
  * [53] J. T. Folsom-Kovarik, G. Sukthankar, and S. Schatz, “Tractable pomdp representations for intelligent tutoring systems,” vol. 4, apr 2013.
  * [54] E. Lukyanchikova, N. Askarbekuly, H. Aslam, and M. Mazzara, “A case study on applications of the hook model in software products,” Software, vol. 2, no. 2, pp. 292–309, 2023.
  * [55] A. Fabijan, P. Dmitriev, H. H. Olsson, and J. Bosch, “The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale,” in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pp. 770–780, IEEE, 2017.
  * [56] M. T. Chi and R. Wylie, “The icap framework: Linking cognitive engagement to active learning outcomes,” Educational psychologist, vol. 49, no. 4, pp. 219–243, 2014.
  * [57] R. Ajjawi, J. Tai, T. L. Huu Nghia, D. Boud, L. Johnson, and C.-J. Patrick, “Aligning assessment with the needs of work-integrated learning: The challenges of authentic assessment in a complex context,” Assessment & Evaluation in Higher Education, vol. 45, no. 2, pp. 304–316, 2020.
  * [58] Y. Zhao, M. Wehmeyer, J. Basham, and D. Hansen, “Tackling the wicked problem of measuring what matters: Framing the questions,” ECNU Review of Education, vol. 2, no. 3, pp. 262–278, 2019.
  * [59] R. W. Picard, S. Papert, W. Bender, B. Blumberg, C. Breazeal, D. Cavallo, T. Machover, M. Resnick, D. Roy, and C. Strohecker, “Affective learning—a manifesto,” BT technology journal, vol. 22, no. 4, pp. 253–269, 2004.

##  Appendix A Appendix A: Quality Assessment Score

In addition to analysing the papers, we also attempted to assess the overall
quality of papers in regards to clarity of motivation behind each paper,
description of the proposed solution, and presence of evaluation section. We
also gave each paper a subjective mark of how relevant it is to the purposes
of our review.

We used the following questions to establish the quality score for each paper:

1) Was the motivation and intended use case for the development of the
recommender system clearly specified?
\- 1 point if clearly stated;
\- 0.5 points if provided, but could be further elaborated;
\- 0 points if hard to identify or not mentioned;

2) Are the user model, domain model, recommendation algorithm, and evaluation
procedure clearly described and reproducible?
\- 1 point if possible to replicate;
\- 0.5 points if described , but difficult to replicate because of lack of
details;
\- 0 points if not described or not clear;

3) Was the proposed recommender system objectively evaluated?
\- 1 point if conducted a fair and unbiased review of their algorithm or
performed a critical analysis of its results;
\- 0.5 points if performed an analysis, but partially biased or not clear or
not critical enough;
\- 0 points if did not conduct a fair and unbiased analysis or did not
critically analyse the results.

4) Was the paper useful in regards to addressing the problem of assessing
learning outcomes in recommender systems.
\- 1 point if the paper directly addresses the assessment of learning outcomes
in RS;
\- 0.5 points if the paper is partially or indirectly addresses the assessment
of learning outcomes in RS;
\- 0 points if the paper is not relevant to the assessment of learning
outcomes in RS.

The maximum possible score is 4 points. After calculating the score for each
of the analyzed papers, we also calucated the average score for all papers and
for each of the three databases (IEEE, ACM, and Springer).

The average quality score for papers is 2.804 out of 4.0. If we examine the
average quality score by source (Fig. 12), we can see that IEEE and ACM
Digital Library have a similar score, while Springer is slightly below.

Figure 12: Average quality score by database

Most importantly, this examination confirmed the overall high quality of the
selection. For more details to see the quality score for each paper, please
refer to our repository 222https://github.com/nurlingo/slr-
ers/blob/main/data.csv.

##  Appendix B Descriptive summaries of the selected papers

This appendix contains a short summary for each of the reviewed papers. For
conveniece, we present the summaries in the same order as the works are
referenced in the SLR.

Fischer [26] used a LOM ontology to automatically generate assessment
exercises based on the relations between concepts. The evaluation was a
questionnaire to students, however not much detail is provided.

Parvez and Blank [27] proposed a pedagogical framework to incorporate learning
style when personalizing feedback for programming tasks. It is one of the
early works advocating the importance of learning style.

Abel et al [28] recommend threads in a discussion forum. They developed a
loosely coupled recommendation module that can be integrated into a discussion
forum of an e-learning system. They proceeded to ask how much data is required
for collaborative filtering, what type of user feedback to consider, and
whether user preferences change over time. At the end, the authors propose an
interesting algorithm, where a different recommendation strategy is used based
on how much data is available for the learner. They evaluate their system
using standard precision recall metrics and an undisclosed internal dataset.

Hoic-Bozic et al [29] use a RS in the context of a university course to
recommend optional collaborative activities and tools. They evaluate the
system by comparing the formal assessment results between the years.

Myers and Chatlani [30] uses Category-based Intrinsic Motivation algorithm to
generate personalized learning content for coding literacy tasks. The learner
model keeps track of user’s correct and incorrect answers, and then uses a
logistic function to calculate a learning score for each skill, reflecting the
user’s current understanding level.

Samin and Azim [31] use generative machine learning to represent user and
domain through a probabilistic topic model, and then calculate the distance
between topic models to make recommendations. The advantage of such approach
is that the recommendations can be made on the basis of a querry in natural
language.

Dahdouh et al [32] use association rules mining to recommend courses. The main
idea behind the method is to find frequent combinations of courses that go
together. The method is quite straightforward and only requires course
enrollments history. The authors concentrate on the architecture and system
implementation detail, however there is practically no evaluation for the
method itself as they only compare the time performance of various algorithms
that do the mining.

Pariserum Perumal et al [33] recommends an e-learning content by finding
patterns in learner behavior. In particular, they review the history of
visited items by topics, and then conclude the learner’s interests based on
that. The paper attempts to take into account the ”dynamic user interests” by
using fuzzy logic and splitting the history into windows, instead of treating
as one whole. This can be useful in an e-learning context, where a user’s
interests might evolve as they learn new things. For evaluation, they measure
the relevance through the Harmonic mean. However, the learning outcomes and
their achievement is not touched in the paper at all.

Cardenas-Cobo et al [34] recommend Scratch programming tasks to university
students that are appropriate to their level. They use traditional CF
approach, where each student has the set of solved exercises with the
corresponding ratings tuple (taste, complexity). Based on the ratings three
types of similarity between users are computed, and then a set of similar
users is produced. The study implements a thorough evaluation methodology,
where both a qualitative questionnaire (including RS performance, user-centric
effects, effects of learning) and a quantitative academic performance data are
used.

Yanes et al [35] recommend course improvement actions to instructors based on
the course description, learning outcomes and assessment results. They frame
it as a classification problem, and compare several ML algorithms to find the
significant features and best performing method. Despite commenting on the
importance of the outcome-based education, they do not measure how their
recommendations affects the achievement of COs.

Fernández-García et al [36] describe the full procedure of creating an ML-
based RS to help students choose academic subjects. The work also serves as a
great tutorial to pre-processing educational data, and describes all taken
steps in a clear reproducible way. Importantly, it demonstrates how feature
engineering and resampling can partially compensate the imbalance and scarcity
in data. However, the recommender system aspect is not significantly
highlighted, and the paper is more of a study on ML classification and data
pre-processing.

Lin and Chen [37] integrated a deep learning-based recommender into a
programming course to recommend course activities. They split student into a
control and experiment groups and conducted a pre- and post-tests to measure
the effect of using the recommender on the learning outcomes of the experiment
group.

Vedavathi and Anil Kumar [38] recommend web resources to students in
university. They used a Deep Recurrent Neural Network and an improved whale
optimization algorithm to classify the e-learners and order their ratings in
sequence.

Amane et al [39] recommend courses to university students using dynamic
ontologies to represent the learners, courses, and links between them. Among
other properties, they take into account languages preferences and learning
style. Their method is an interesting hybrid where knowledge-based approach is
used to conduct clustering of similar learners. The dynamicity in their
approach means that they update the ontologies on regular basis instead of
calculating them once in the very beginning.

Belver et al [40] propose a requirements elicitation technique in the form of
theatre that can help to extract a broader learning context including
affective dimensions. Then they design an ontology that encapsulates these
learner and context properties including learning style, mood, and even
weather. They make recommendations by finding the most similar learner context
accross these many dimensions.

Okubo et al [41] recommend revision materials to university students. The
recommendations are based on the quiz results, whereby the system suggests
lecture slides that are semantically similar to the questions where the
learner made mistakes. As their evaluation, they split students into control
and experiment groups and used a pre and post-test to see which group performs
better on average.

Long and Aleven [42] describe an ITS for learning algebraic equations, which
uses Bayesian Knowledge Tracing (BKT) to model a student’s mastery of skills
over time and select appropriate exercises. They compare the ITS against a
popular educational game from the same domain in a split study, and then
evaluate the learning outcomes and learners’ enjoyment for each system.
Interestingly, their results indicate that the enjoyment does not necessarily
correlate with the actual learning. However, it does lead to greater
engagement.

Fazeli et al [43] note that evaluating RS from the point of view of accuracy
is not sufficient, and it should be supplemented with a qualitate set of user-
centric metrics. They also run an experiment to demonstrate that graph-based
CF can outperform user-based CF in regards to these qualitative metrics. The
work is important, as it questions the validity of precision as the default
metric to measure effectiveness of RS in the context of education.

Moradi et al [44] recommend educational resources to children using mobile
technology and by observing the group dynamics. They model interaction between
children as a directed acyclic graph (DAG), and assign subject topics to each
child. Then based on the interaction data, they suggest educational resource
in the topics which were labelled as ’weak’. They evaluate the method using
simulation to see how fast the correct information will disseminate in the
DAG. Importantly, they recognize that children learn from each other, and by
suggest that by influencing the most active nodes in the graph an educator can
improve the learning for the whole group.

Niemann and Wolpers [45] propose a way to calculate similarity between
learning objects implicitly based on what items were used together in one
session. They argue that this can be an alternative to the content-based
approach when metadata on the learning objects is not available or difficult
to gather. The main idea behind their approach is that ”users’ knowledge and
context is inherent in their activities”, therefore based on the co-usage of
certain objects, one can derive the correlation between them.

Bustos López et al [46] suggest a CF-based recommender that incorporates
emotion detection results into the ratings, and compare it to a traditional CF
algorithm. The evaluation includes user-centric questionnaire in addition to
F1 metrics, which provides us with both a quantitative and qualitative
perspectives. They also make a weighted matrix to compare the features with
other similar systems. Though it is not clear why a system with most features
should necessarily be superior, however, having the set of commonly-occurring
features is of value in itself.

Joy et al [47] propose a generic ontology for e-learning that combines the
learner, learning objects, and learner’s path into a single model. The
ontology allows to recommend resources even for new users as the article’s
main goal is to solve the pure cold start problem. The learner model includes
personal information and learning styles, however there are no details on the
privacy aspect of the user data. The evaluation is done by gathering real
users’ satisfaction with the recommendations on 1 to 5 scale.

Pereira et al [48] propose to extract learner data from social network
profiles. They argue that social networks are a much richer source of data,
rather than traditional LMS. The authors use entities, concepts, and keywords
detection and domain ontologies to extract user interests from Facebook posts
and groups. They let users confirm the correctness of the extracted
information. The resources are recommended from several sources including Open
University learning objects, DBpedia and Youtube. To evaluate the
recommendations they conducted a case study and used questionnaires to ask the
participants about the relevance of extracted interest and recommended items.

Wu et al [49] propose an approach where both learning activities and learner
profiles are represented as fuzzy trees. Various techniques are applied to
calculate similarity be between learning activities based on categories
assigned to them. The fuzzy tree combinations are used to derive the fuzzy
tree of all categories learned by a learner. Moreover, the model facilitates
for a manual specification of prerequisite relation between activities and
also automatic generation of precedence relations from the sequences in
learners’ histories. The learner profile contains learner preference scores
for available categories in the category tree, and the scores are used to
calculate similarity between users and to match a learner with an activity. A
hybrid recommendation method is used, including the knowledge of relations
between the items and users’ preferences and learning goals, and the use of
collaborative technique that takes into account ratings by similar users.

Intayod et al [50] developed a reinforcement learning agent to recommend
learning objects to students. As their algorithm, they suggested a
modification of a multi-arm bandit. Their aim was to maximize clickthrough
rate.

Liu et al [51] proposed a method to model the learners, resources, exercises,
and knowledge concepts within a single heterogeneous graph. The graph includes
the learner interactions with resources and exercises, and maps the exercises
to knolwedge concepts. After a few transformations it can serve as the
training data for deep learning methods.

Montuschi et al [52] help job seekers to acquire missing skills for a
particular vacancy. The identify a gap between the user and his desired job,
and then formulate it as desired learning outcomes (LOs). Then they use a
semantic taxonomy to map the LOs to the educational resources. The system is a
classic knowledge-based recommender system, however authors themselves do not
refer to it as such. Interestingly, they allow users to participate in mapping
resources to LOs, thus using a form of crowdsourcing to augment the system’s
knowledgebase.

Folsom-Kovarik et al [53] proposed an intelligent tutoring system based on
Partially Observable Markov Decision Process. The system models learner as a
set of knowledge states each representing a desired learning outcome that
learner needs to cover. Then the system suggests corrective actions to
increase the probability of each state being covered and not leaving gaps.

Nursultan Askarbekuly Nursultan Askarbekuly is a PhD student at the Institute
of Software Engineering, Innopolis University and at the Faculty of
Organizational Sciences, University of Belgrade. His main research interest is
data-driven educational technology, empirical software engineering, and
product development.
---
Ivan Luković Ivan Luković is a Full Professor at University of Belgrade,
Faculty of Organizational Sciences. His main research interests are focused on
Theory of data models, System design, particularly logical and physical
database design, and development and usage of MDSD / CASE tools in Software
Engineering and System Design.
---