# A high performance computing method for accelerating temporal action
proposal generation

Shiye Lei  School of Automation Science and
Electrical Engineering, Beihang University
Beijing, China
<EMAIL_ADDRESS>Tian Wang  School of Automation Science and
Electrical Engineering, Beihang University
Beijng, China
<EMAIL_ADDRESS>Youyou Jiang  School of software
Tsinghua University.
Beijing, China
<EMAIL_ADDRESS>Zihang Deng  School of Automation Science and
Electrical Engineering, Beihang University
Beijing, China
<EMAIL_ADDRESS>Hichem Snoussi  Institute Charles Delaunay-LM2S FRE CNRS
2019
University of Technology of Troyes
Troyes, France
<EMAIL_ADDRESS>Chang Choi  IT Research Insitute
Chosun University
Gwangju, Republic of Korea
<EMAIL_ADDRESS>
###### Abstract

Temporal action proposal generation, coming from temporal action recognition,
is an important and challenging problem in computer vision. Because of the big
capacity of video files, the speed of temporal action recognition is difficult
for both researchers and companies. To training a convolutional neural network
(CNN) for temporal action recognition, a lot of videos are required to put
into the CNN. A speed-up for the task should be proposed for the training
process to achieve the faster response of temporal action recognition system.
To address it, we implement ring parallel architecture by Massage Passing
Interface (MPI). Different from traditional parameter server architecture,
total data transmission is reduced by adding a connection between multiple
computing load in our new architecture. Compared to parameter server
architecture, our parallel architecture has higher efficiency on temporal
action proposal generation task with multiple GPUs, which is significant to
dealing with large-scale video database. And based on the absence of
evaluating time consumption in a distributed deep learning system, we proposed
a concept of training time metrics which can assess the performance in the
distributed training process.

###### Index Terms:

Video processing, Temporal action proposal generation, Distributed deep
learning

##  I Introduction

With the rapid development of the Internet and camera, the number of videos is
increasing at a very high speed. There are millions of video submissions on
video-sharing websites like YouTube every day. Besides, the video surveillance
system plays an important role in maintaining security [1][2][3]. These video
files contain a lot of information for human, such as time duration and action
classify[4][5]. How to make full use of videos is an indispensable step for
building a smart city. It is vital for the development of information age to
extract information from a large number of videos by automatically. Action is
the most important information for videos because the essence of the video is
recording varieties of motion. So a significant branch of video task is action
recognition, which aims to recognize the class of action from a trimmed video.
But the task is limited because its research object is videos that have been
manually trimmed and only contain single action. The majority of videos in the
real world are untrimmed videos and contains multiple action instances in a
single video. The problem requires another challenging task: temporal action
detection, which aims to recognize the temporal boundaries and classes of
action instance from untrimmed videos.

Temporal action detection usually includes two steps: proposal and
classification. Proposal stage focuses on detecting action boundary and
generating action instance with untrimmed video. Classification is aim to
recognize the class of action instance produced in the previous step. For the
task of temporal action detection, classification has achieved high accuracy.
And the speed and precision of proposals is the main factor limiting temporal
action detection [6][7].

High-quality proposals should meet two requirements [8]: (1) high recall; (2)
high overlap with ground truth. And a good algorithm of generating proposal
should not only generate excellent proposals, but its speed should be as fast
as possible. Because videos occupy a large amount of memory, we must improve
the speed of method so for being applied to practice.

Figure 1: The framework of our approach. (a) Two-stream network is used to
encode visual features in snippet-level. (b) The architecture of Boundary
Sensitive Network

Most proposal generation methods generate proposals by using sliding windows
[9][10][11][12]. But the pre-defined durations and intervals of sliding
windows cannot generate proposals with flexible length, which greatly reduced
the precision of proposals. Boundary Sensitive Network (BSN) [8] used a
temporal network with 3 convolution layers to deal with video feature
sequences and could generate proposals with flexible duration. Though BSN has
a good performance, its multiple steps and networks make the speed not
optimistic.

Recently, lots of high performance computing methods are introduced into the
field of compute vision for acceleration. Distributed computing is one of the
more common methods. It is committed to increasing the number of devices in
exchange for speeded-up. Parameter Server is a famous acceleration method for
training deep learning network. On this framework, Multiple workers compute
different input data and central server integrates the outputs from workers,
which is very simple and effective. But when the number of workers has
continuous growth, which leads to a significant increase in communication
volume, acceleration will get worse.

To address these problems and improve the speed of producing proposals, we
designed a new parallel computing framework. Because BSN is the state-of-art
method for temporal action proposal generation, so our new framework was
tested on BSN and had a higher efficiency compared to the popular Parameter-
Server Framework[13].

In summary, the main contribution of our work is two-fold:

  1. (1)

We build a new parallel computing framework to speed up temporal convolution
network on temporal action recognition with high efficiency.

  2. (2)

A metric is put forward to evaluate the time consumption in the distributed
deep learning field.

##  II Related work

Temporal action detection aims to detect action instances from the untrimmed
video. The task could be divided into two steps: proposal and classification.
Though some methods do the two steps at the same time, the majority of methods
take the task as a serial process and finish proposal and classification
separately.

Temporal action proposal generation. Proposal generation is the distinct
characteristic of Temporal action recognition. Proposal generation aims to
detect the start and end boundary of action instance in the untrimmed video.
Earlier methods used sliding windows to generate proposals [14][15]. Then some
algorithms [9][16][10][11] began to pre-define temporal duration and intervals
of proposals, and evaluated them with multiple methods like recurrent neural
network (RNN) and dictionary learning. Another popular method for proposal
generation is TAG [17], which utilized watershed algorithm to do the project.
Though TAG can generate proposals with flexible boundaries and durations, it
is lack of evaluation to these proposals. BSN [8] has a good performance of
generating proposals, which is benefit from its temporal convolution network.
By generating a possibilities sequences and selecting candidate proposals, it
can achieve a high recall. But a series of CNNs slow down the speed and speed
make it difficult to be applied to practice.

Distributed deep learning Because deep learning has a wide range of
application, acceleration is significant to let it more widely to be used.
Distributed deep learning, which is based on parallel computing, belongs to
high performance computing and accelerate CNN by using more machines like
GPUs.

MapReduce[22] was proposed by Google and dissemble compute into map and
reduce, which divided compute into tow steps of Map and Reduce. But it has a
strict requirement of consistency. To address the problem, Graphlab[23] used
an abstract way like the image to communicate, which also lead to low
scalability. Jeff Dean proposed Parameter-Server Framework (PS) [13], which
uses a parameter server to store the newest weight parameters of CNN. When the
number of GPUs increases, the efficient of PS will have a great decline
because of the big communication.

Video task is closed to real life and Applied in practice is its final goal.
So besides accuracy, speed of methods is also an important indicator. Based on
these, our method is superior to others in acceleration for temporal action
proposal generation.

##  III parallel computing acceleration

###  III-A classes of parallel computing

For training of the neural network, GPU is a much faster platform than CPU
because of its architectural advantage on matrix computation. So we used GPU
as the computing platform for our temporal network. Further, we applied
parallel computing based on GPU to accelerate our network for more powerful
capabilities of processing video. The hard architecture of GPU is shown as
Fig. 2. Stream Processor (SP) is the basic computing unit of GPU and Stream
Multiprocessor (SM) is composed of a certain number of SM, register, share
memory and L1/L2 cache. From the picture, we can see that one SM contains
multiple SP but only one instruction unit. So for single SM, it only supports
single instruction multiple data (SIMD) but not multiple instruction multiple
data (MIMD). When GPUs are used to train CNN models, parameters of models are
stored at device memory, which decided that the form of parallel computing of
GPU in deep learning field is SIMD. After determining SIMD, the parallel
architecture of GPU can be divided into two classes of model parallelism and
data parallelism.

Figure 2: Hard architecture of GPU

###  III-B model parallelism

Model parallelism means different machines (GPU or CPU) in a distributed
system are responsible for different parts of a network model. For example,
different layers in a neural network model or different parameters in the same
layer are assigned to different machines. The structure of model parallelism
is shown as Fig. 3. In general, the reason for applying model parallelism is
oversized for the neural network so that single GPU cannot load the whole
model.

Figure 3: Model parallelism

###  III-C data parallelism

Data parallelism means the input data are divided into several parts and
delivered to different machines. There is a complete model in each machine and
these machines run the same program to deal with allocated data. Training CNN
is a serial process, i.e., only after computing the gradients for current data
and upgrading parameter weights, the next data can be put into the machine.
The key of model parallelism is that all of GPUs have the same CNN model,
which we called them model replicas. But the data for each GPU is different.
We integrate different weight gradients $\nabla w$ calculated by all of GPUs
and upgrade parameters of the model.

With its simple and understandable structure, Parameter Server (PS) becomes
the main data parallelism framework and got support from some mainstream deep
learning framework like TensorFlow [24]. The architecture of PS is shown as
Fig. 4, where $\Delta w$ is the weight gradients computed by model replicas
like GPUs and $w^{\prime}$ is the newest weight parameters. PS stores the
parameters of the model. Model replicas compute different parameter weights
and then upgrade the parameters in the parameter server.

Figure 4: Parameter Server framework

###  III-D ring parallel architecture

From Fig. 4 we can see that the communications volumes increase linearly with
the increasing number of GPUs. We suppose the size of CNN model is $M$ and $N$
GPUs are used in our distributed system, so the communications volume is
$N\cdot M$. If the number of GPUs achieved a high level, the large
communications volume will greatly limit the training speed of CNN model.

To address the problem of large communication volume in distributed deep
learning system, we proposed ring parallel architecture. By building
communication between GPUs with Message Passing Interface (MPI), our ring
parallel architecture can reduce the pressure of communication. The ring
parallel architecture is shown as Fig. 5. We changed the parallel topology to
ring and averaged the communications volumes. The ring architecture upgrades
weights through two steps including scatter and gather.

Figure 5: Ring parallel architecture. Multiple colors denote the weight
gradient computed by each GPU is different.

####  III-D1 scatter

We divide weights in every GPU into $N$ parts, where $N$ is the number of GPU
utilized in the architecture. After all of GPUs got different weight gradients
by computing different input data, like there are different colors in Fig. 5
and a row of colored blocks denotes a part. The $n$-th GPU passes its own
$(n-i)\%N$-th block of weight gradients to its right neighbor and receives
$(n-i-1)\%N$-th block of weight gradients from its left neighbor, where $i$ is
the round of scatter. Fig. 6 shows the detail after one round of scatter.

Figure 6: Scatter. In the scatter step, the GPU passes a row of weight
gradient (all of the colors in this row) to the same position in its next GPU.

After $N-1$ rounds of scatter, $n$-th GPU has collected $(n+1)\%N$-th block of
weight gradients from all GPUs, which is shown as Fig. 7. After scatter, each
GPU has a block of gradients which is from all GPUs.

Figure 7: Distribution of weight gradients after scatter. There is a row in
every GPU that has collected weight gradients from all of GPUs (all of the
colors).

####  III-D2 gather

Like scatter, GPUs also pass a block of weight gradients to the next GPU in
the process of gather. Through $N-1$ rounds of gather, the $(n+1)\%N$-th block
of weight gradients in the $n$-th GPU is passed to all of other GPUs. In the
$i$-th round of gather, the $n$-th GPU passes its own $(n-i-1)\%N$-th blocks
of weight gradients to its right neighbor and receives the $(n-i-2)\%N$-th
blocks of weight gradients from its left neighbor. Different from scatter,
GPUs don’t need to add but replace its own block by the block received. After
gather, we can see that all of GPUs have obtained all weight gradients
computed by every GPU, which is shown as Fig. 8.

Figure 8: Distribution of weight gradients after gather. All of the weight
gradients are merged in each GPU.

###  III-E training time metrics

To explore the relationship between the number of GPUs and training time and
evaluate parallel architecture, we defined training time metrics $T(n)$, where
$n$ is the number of GPU used.

Training time in distributed deep learning system could consist of three
parts:(1) $t_{1}$ for forward propagation and backward propagation of single
GPU; (2) $t_{2}$ for communication of weight gradients between GPUs or between
GPU and CPU; (3) $t_{3}$ for preparation before training process and finishing
work after training.

For Parameter Server framework, $t_{1}$ is inversely proportional to the
number $n$ of GPUs used. $t_{2}$ is proportional to the number $n$ of GPUs and
$t_{3}$ has nothing to do with $n$. So the training time metrics for PS
framework $T_{PS}(n)$ is shown as below:

$t=\frac{T}{n}+C\cdot n+P,\ n=2,3,\dots$ (1)

Where $T$ is the training time with using single GPU, $C$ is the communication
time and $P$ is the preparation time for opening and closing deep learning
platform.

For our ring parallel architecture, $t_{1}$ is also inversely proportional to
the number of $n$ of GPUs used. Let the size of $\Delta w$ in each GPU is $K$,
single GPU send $\frac{K}{n}$ to his right neighbor each round. Every GPU do
$n-1$ rounds of scatter and $n-1$ rounds of gather, so the total communication
volume is $2K\cdot\frac{n-1}{n}$. Then we can get that $t_{2}$ is proportional
to $\frac{n-1}{n}$. And $t_{3}$ is also a constant. So the training time
metrics for ring parallel framework $T_{Ring}(n)$ is shown as below:

$t=\frac{T}{n}+C\cdot\frac{n}{n-1}+P,\ n=2,3,\dots$ (2)

For a distribute deep learning system, especially a large system, total time
consumption $t$ depends largely on the csommunication time $t_{2}$. The most
difference between these two training time metrics is $t_{2}$. As the number
of GPUs $n$ increases, $t_{2}$ in ring parallel architecture will be smaller
than PS framework.

##  IV experiments

Dataset. ActivityNet-1.3 [6] is a normal video dataset for the temporal action
proposal generation task. It contains 19994 untrimmed videos containing 200
classes of action instance and corresponding annotations. each untrimmed video
includes one or more action instances. ActivityNet-1.3 is divided into
training set, test set and validation set in a ratio of approximately 2:1:1.
In this part, we will compare the performance of various commonly used methods
for temporal action detection on ActivityNet-1.3.

Temporal action proposal generation. For lots of tasks in video analysis, they
do not handle video directly but deal with video feature sequence. Video
feature sequence is usually encoded by neural network with special structure.
In this paper, we used two-stream network [25] as an encoder to transfer video
into a set of vectors.

Implementation details. Two-stream network [25] whose temporal network is BN-
Inception [27] and spacial network is ResNet [28] is used to encode videos and
generate video feature sequences. Then Boundary Sensitive Network (BSN) shown
in Fig. 1(b) deals with these video sequences and produces final action
proposals. We used both Parameter Server and Ring Architecture to run temporal
action proposal generation task and compare their time consumption. We
implement our distributed neural network with TensorFlow [24]. Our parallel
computing platform are equipped with 8 TITAN V-100 GPUs.

Because of occupying huge memory and a large number of video files,
distributed deep learning is indispensable for applying temporal action
detection algorithm to the actual. Based on that the inefficient of the
traditional PS framework on distributed deep learning, we applied parallel
ring architecture to our temporal convolution network and received a good
result. The speed ratio with these two parallel frameworks is shown as Fig. 9.
In the figure, $Speedratio=t_{0}/t$, where $t_{0}$ is the time cost with
single GPU. As the number of GPU increases, the performance of parallel ring
architecture is getting better and better than PS architecture.

Figure 9: Comparison of speed ratio between PS and parallel ring architecture

The reason for efficiency improve of Ring Architecture is the decrease of
communication volume, which is obvious. To further explore the relationship
between the number of GPUs and training time, we defined training time
function $T(n)$, where $n$ is the number of GPU used.

In order to test the validation of our training time metrics, we use the
number of GPUs used, $n$, as the independent variable and the training time as
the dependent variable to fit the Eq. 1. The fitting curve is shown in Fig.
10(a) and we can get that $T=4223.8,C=12.1,P=290.8$.

(a) Parameter Server architecture

(b) Parallel ring architecture

Figure 10: Fitting curve of training time function

If we use the same training time metrics 1 to fit the training time with using
parallel ring architecture, we would get the result of $C=-3.8<0$, which is
obviously unreasonable. The fitting curve for ring parallel architecture with
using Eq. 2 is shown in Fig. 10(b) and we can get that
$T=4400.1,C=59.6,P=363.5$.

The parameter $C$ is much bigger in parallel ring architecture than it in PS,
which indicates that parallel ring architecture is lower than PS in transfer
speed. But if the number of GPUs increases to a obvious level, especially in
large scale deep learning like some video task, parallel ring architecture
will be a better choice than Parameter Server framework.

##  V conclusion

In this paper, we applied a new parallel architecture, ring parallel
architecture, to accelerate our network by reducing the pressure of
communication. Our method have a higher acceleration efficiency compared with
other state-of-art proposal generation methods, which is significant for
dealing with large-scale video database in industrial filed. And we also
proposed metrics to evaluate the time consumption of an distributed deep
learning system and proved its effectiveness.

## References

  * [1] K. Muhammad, R. Hamza, J. Ahmad, J. Lloret, H. Wang, and S. W. Baik, “Secure surveillance framework for IoT systems using probabilistic image encryption,” _IEEE Transactions on Industrial Informatics_ , vol. 14, no. 8, pp. 3679–3689, Aug 2018.
  * [2] M. Sajjad, I. U. Haq, J. Lloret, W. Ding, and K. Muhammad, “Robust image hashing based efficient authentication for smart industrial environment,” _IEEE Transactions on Industrial Informatics_ , pp. 1–1, 2019\.
  * [3] T. Wang, M. Qiao, Z. Lin, C. Li, H. Snoussi, Z. Liu, and C. Choi, “Generative neural networks for anomaly detection in crowded scenes,” _IEEE Transactions on Information Forensics and Security_ , vol. 14, no. 5, pp. 1390–1399, 2018.
  * [4] K. Muhammad, S. Khan, V. Palade, I. Mehmood, and V. H. C. De Albuquerque, “Edge intelligence-assisted smoke detection in foggy surveillance environments,” _IEEE Transactions on Industrial Informatics_ , pp. 1–1, 2019.
  * [5] T. Wang, Z. Miao, Y. Chen, Y. Zhou, G. Shan, and H. Snoussi, “Aed-net: An abnormal event detection network,” _arXiv preprint arXiv:1903.11891_ , 2019\.
  * [6] F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles, “Activitynet: A large-scale video benchmark for human activity understanding,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 961–970.
  * [7] Y. G. Jiang, J. Liu, A. R. Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar, “Thumos challenge: Action recognition with a large number of classes,” 2014.
  * [8] T. Lin, X. Zhao, H. Su, C. Wang, and M. Yang, “Bsn: Boundary sensitive network for temporal action proposal generation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 3–19.
  * [9] S. Buch, V. Escorcia, C. Shen, B. Ghanem, and J. Carlos Niebles, “Sst: Single-stream temporal action proposals,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2911–2920.
  * [10] F. Caba Heilbron, J. Carlos Niebles, and B. Ghanem, “Fast temporal activity proposals for efficient detection of human actions in untrimmed videos,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1914–1923.
  * [11] V. Escorcia, F. C. Heilbron, J. C. Niebles, and B. Ghanem, “Daps: Deep action proposals for action understanding,” in _European Conference on Computer Vision_. Springer, 2016, pp. 768–784.
  * [12] Z. Shou, D. Wang, and S.-F. Chang, “Temporal action localization in untrimmed videos via multi-stage cnns,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1049–1058.
  * [13] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le _et al._ , “Large scale distributed deep networks,” in _Advances in neural information processing systems_ , 2012, pp. 1223–1231.
  * [14] S. Karaman, L. Seidenari, and A. Del Bimbo, “Fast saliency based pooling of fisher encoded dense trajectories,” in _ECCV THUMOS Workshop_ , vol. 1, no. 2, 2014, p. 5.
  * [15] L. Wang, Y. Qiao, and X. Tang, “Action recognition and detection by combining motion and appearance features,” _THUMOS14 Action Recognition Challenge_ , vol. 1, no. 2, p. 2, 2014.
  * [16] J. Gao, Z. Yang, K. Chen, C. Sun, and R. Nevatia, “Turn tap: Temporal unit regression network for temporal action proposals,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 3628–3636.
  * [17] Y. Zhao, Y. Xiong, L. Wang, Z. Wu, X. Tang, and D. Lin, “Temporal action detection with structured segment networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2914–2923.
  * [18] H. Wang, A. Kläser, C. Schmid, and L. Cheng-Lin, “Action recognition by dense trajectories,” in _CVPR 2011-IEEE Conference on Computer Vision & Pattern Recognition_. IEEE, 2011, pp. 3169–3176.
  * [19] C. Feichtenhofer, A. Pinz, and A. Zisserman, “Convolutional two-stream network fusion for video action recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1933–1941.
  * [20] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool, “Temporal segment networks: Towards good practices for deep action recognition,” in _European conference on computer vision_. Springer, 2016, pp. 20–36.
  * [21] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 4489–4497.
  * [22] J. Dean and S. Ghemawat, “Mapreduce: simplified data processing on large clusters,” _Communications of the ACM_ , vol. 51, no. 1, pp. 107–113, 2008\.
  * [23] Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. M. Hellerstein, “Distributed graphlab: a framework for machine learning and data mining in the cloud,” _Proceedings of the VLDB Endowment_ , vol. 5, no. 8, pp. 716–727, 2012.
  * [24] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin _et al._ , “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” _arXiv preprint arXiv:1603.04467_ , 2016.
  * [25] K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in _Advances in neural information processing systems_ , 2014, pp. 568–576.
  * [26] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [27] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” _arXiv preprint arXiv:1502.03167_ , 2015.
  * [28] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.


# PEEK: A Large Dataset of Learner Engagement
with Educational Videos

Sahan Bulathwela1, María Pérez-Ortiz1, Erik Novak2, Emine Yilmaz1 and John
Shawe-Taylor1<EMAIL_ADDRESS>${}^{1}\ $Centre for Artificial
Intelligence, University College London (UK) ${}^{2}\ $Jožef Stefan Institute,
Jožef Stefan International Postgraduate School (Slovenia)

(2021)

###### Abstract.

In this work, we release a large and novel dataset of learners engaging with
educational videos in-the-wild. The dataset, named Personalised Educational
Engagement with Knowledge Topics (_PEEK_), is one of the first publicly
available datasets that address personalised educational engagement.
Educational recommenders have received much less attention in comparison to
e-commerce and entertainment-related recommenders, even though efficient
personalised learning systems could improve learning gains significantly. One
of the main challenges in advancing this research direction is the scarcity of
large, publicly available datasets. In the PEEK dataset, educational video
lectures have been associated with Wikipedia concepts related to the material
of the lecture, thus providing a humanly intuitive taxonomy. We believe that
granular learner engagement signals, in unison with rich content
representations, will pave the way to building powerful personalisation
algorithms that will revolutionise educational and informational
recommendation systems. Towards this goal, we 1) construct a novel dataset
from a popular video lecture repository, 2) identify a set of benchmark
algorithms to model engagement, and 3) run extensive experimentation on the
PEEK dataset to demonstrate its value. Our experiments with the dataset show
promise in building powerful informational recommender systems. The dataset
and the support code is available at https://github.com/sahanbull/PEEK-
Dataset.

††copyright: iw3c2w3††journalyear: 2021††conference: ORSUM ’21: 4th Workshop
on Online Recommender Systems and User Modeling at ACM RecSys 2021; October
02, 2021; Online††booktitle: ORSUM ’21: 4th Workshop on Online Recommender
Systems and User Modeling at ACM RecSys 2021, October 02, 2021, Online††ccs:
Information systems Users and interactive retrieval††ccs: Information systems
Personalization††ccs: Information systems Recommender systems††ccs:
Information systems Information extraction††ccs: Applied computing Interactive
learning environments

##  1\. Introduction

Developing artificial intelligence systems that, mildly at least, understand
the structure of knowledge is foundational to building effective
recommendation systems for lifelong education(Pardos et al., 2014; Bulathwela
et al., 2020d; Yu et al., 2020), as well as for many other applications
related to knowledge management and tracing(Lewis et al., 2020; Yano and Kang,
2016). In the context of personalised education, the research communities have
tirelessly worked on building Intelligent Tutoring Systems (ITS) which have
been their focus from the early applications of AI in education(Corbett and
Anderson, 1994). ITS focus heavily on personalising testing opportunities that
will allow learners to demonstrate their mastery of a Knowledge Component
(KC), an atomic unit of knowledge that can be learned and mastered. These
systems are designed for learning experiences with limited scope (such as
learning about a specific topic, short course, etc.) which makes hand crafting
KCs and encouraging repetitive test taking operationally feasible. More recent
developments in online education has led to the boom of Open Educational
Resources (OERs) and Massively Open Online Courses (MOOCs) moves from ITS to
educational recommendation systems and personalised e-learning platforms that
can cater a more informal learning setting where self motivated learners
discover educational materials online to pursue their lifelong learning goals.
To succeed in this landscape, a wider range of factors such as the diversity
of learners, their personal drivers and how these drivers change over time has
to be accounted for. One of the major barriers in building next generation
educational recommendation systems is the scarcity of publicly available
datasets.

###  1.1. Our Contribution

We publish _(P)ersonalised (E)ducational (E)ngagement Linked to (K)nowledge
Topics_ (PEEK), a novel dataset that comprises of watch time interactions of
over 20,000 informal learners watching over 10,200 unique educational video
lectures in an OER repository. Our contribution is two-fold, consisting of: i)
constructing a novel dataset to predict learner engagement with educational
videos and ii) formalising a prediction task with several baselines. The video
lectures are partitioned into multiple fragments where the parts are
transcribed and associated with Wikipedia topics using entity linking. PEEK
dataset uses a humanly intuitive content representation where the topics are
atomic Wikipedia pages. Having humanly intuitive content representations
allows building explainable learner models that encourage learner meta-
cognition and self-regulation, a valuable feature of a technology enhanced
learning system(Bull and Kay, 2008). The normalised watch time of individual
users is discretised and provided as a target label defining a binary (engaged
vs. not engaged) prediction task. We further identify several baselines from
prior work and establish a formal task to benchmark predictive performance on
the proposed dataset.

##  2\. Related Work

Publication of this dataset is inspired by advances in multiple research
verticals. PEEK dataset is an educational recommendation dataset that contains
learner interactions with fragments of video lectures. We outline the relevant
works from these different research verticals in this section.

###  2.1. Intelligent Tutoring Systems and Educational Recommenders

_Knowledge Tracing_ (KT) and _Item Response Theory_ (IRT), the foundational
methods used in Intelligent Tutoring Systems, are evolving from traditional
machine learning(Corbett and Anderson, 1994; Yudelson et al., 2013; Vie and
Kashima, 2019; Pavlik et al., 2009; Pelánek et al., 2017) to deep learning
models(Piech et al., 2015; Jiang et al., 2019) that improve predictive
performance by sacrificing interpretability and transparency, crucial
requirements for many of these systems(Barria Pineda and Brusilovsky, 2019;
Bull, 2020). It is also hard to overlook the data hunger of neural approaches
considering the possible data scarcity in an educational setting where
additional information should be exploited. While accompanying these
additional drawbacks, deep learning models used in these tasks are still not
guaranteed to outperform classical approaches(Wilson et al., 2016).

Another major drawback of ITS research is its primary focus into predicting
knowledge mastery via explicit learner feedback that comes through test
taking. Although test taking is a reliable approach to verify learning, this
approach comes with its fair share of limitations. Tests need to be carefully
crafted by experts in a way that the skill mastery of learners on specific
skills can be carefully verified with accompanying hints(Fyfe, 2016a, b; Choi
et al., 2020) costing substantial human capital. Scaling automatic question
generation is an open challenge. The other stakeholder segment put into
pressure by question solving is the learners themselves. Putting the learner
in front of tests too frequently can significantly hinder the user experience
driving them to abandon the system.

Beyond the use of costly explicit user interactions, there are various other
implicit interactions learners execute within an interactive e-learning system
that can be potentially exploited. Learner engagement with educational
materials is also indicated by implicit interactions such as
watching/pausing/rewinding/replaying video lecture, taking part in discussion
fora etc.(Ramesh et al., 2014). In the context of videos, although the
potential of using watch time for engagement prediction(Wu et al., 2018) and
personalisation(Covington et al., 2016) has led to positive results, this idea
is under-explored in the education domain with only a handful of studies
showing promise(Guo et al., 2014; Bulathwela et al., 2020b) in population-
based engagement prediction. Although there is a strong interest in building
state-aware, personalised educational systems in the recent years(Bulathwela
et al., 2021, 2020c), an obvious reason for slow growth of personalised
educational recommenders is the scarcity of publicly available datasets.

Unfortunately, there is very limited work in the Information Retrieval (IR)
community that deals with a personalisation scenario similar to PEEK dataset
as IR scenarios in education are significantly different from general click
behaviours for which datasets are abundantly available. TrueLearn Novel
model(Bulathwela et al., 2020e) is the only recent algorithmic contribution
that deals with a similar dataset where topical contents automatically
extracted from educational videos are used to predict learner engagement.
Apart from sophisticated learner modelling techniques such as TrueLearn Novel,
other basic recommendation strategies, such as content-based filtering and
collaborative filtering, are also applicable to this dataset(Katz et al.,
2011).

###  2.2. Video Fragments

Although majority of studies in the field focus on recommending content items
that are relevant to a learner, the possibility of recommending parts of items
(e.g. a part of a video in contrast to an entire video) has been investigated
lately. Proximity-aware information retrieval that exploits the positional
structure of tokens is not a new idea(Schenkel et al., 2007) in IR. Segmenting
videos and building a table of contents using video segments has proven to be
useful in efficiently summarising video content(Mahapatra et al., 2018;
Google, 2021). Breaking informational videos into fragments has also shown
promise in efficient previewing(Bulathwela et al., 2020a; Pérez-Ortiz et al.,
2021; Chen et al., 2018) and enabling non-linear consumption of videos(Verma
et al., 2021). Our prior proposal, TrueLearn Novel(Bulathwela et al., 2020e)
model demonstrates the potential of using fragment-wise recommendation in
educational recommenders. For these experiments, we created video parts of
5,000 characters (5 minutes) in the fragmentation process. This allows the
e-learning system to have video fragments that contain a satisfactory amount
of knowledge while keeping the video fragment length at a favourable value in
terms of retaining viewer engagement(Guo et al., 2014). While TrueLearn Novel
demonstrates promise, we strongly believe that availability of such a dataset
to the public is critical in pushing the frontiers of research in (fragment-
based) educational recommendation. PEEK dataset addresses this need.

###  2.3. Scalable Feature Extraction

In addition to manual question generation, systems using KT and IRT often
relies on expert labelling of the Knowledge Components (KCs)(Selent et al.,
2016) (sometimes also for the hierarchy of knowledge(Bauman and Tuzhilin,
2018) and defining a Q-matrix(Tatsuoka, 1983)), which is time consuming and
not scalable to lifelong learning applications. There are various machine
learning based unsupervised learning approaches that are used to extract
latent topics from textual content. Approaches such as Latent Dirichlet
Allocation (LDA)(Blei et al., 2003), Latent Semantic Analysis (LSA)(Dumais,
2004) and other probabilistic approaches(Hofmann, 2013; Liang, 2019) are
potential candidates in this area. However, these unsupervised learning
approaches suffer from complex hyper-parameter tuning processes and limited
interpretability of the discovered _latent_ KCs creating gaps in transparency.
A step forward in this area has been using _explicit_ representations based on
taxonomies such as Wikipedia concepts(Gabrilovich et al., 2007; Egozi et al.,
2011).

_Wikification_ , a form of entity linking(Brank et al., 2017; Ferragina and
Scaiella, 2010) has shown great promise for automatically capturing the KCs
covered in an educational resource. This technology is on a promising path
towards providing _automatic_ , _humanly-intuitive (symbolic)_ representations
from Wikipedia, representing at the same time _up-to-date knowledge_ about
_many domains_. Wikipedia, being one of the largest encyclopedias in the
world, evolves with time due to the contributor population that constantly
updates it. Due to these reasons, we use Wikification(Brank et al., 2017) to
generate KCs that are included in the PEEK dataset.

###  2.4. Related Datasets

The majority of datasets that are related to this task come from the
_knowledge tracing_ domain(Corbett and Anderson, 1994; Piech et al., 2015)
which focuses on recovering knowledge/skill mastery of learners based on how
they answer to specific exercises. This approach, as discussed in Section 2,
is much more explicit and effort intense than inferring it using implicit
feedback (e.g. watched patterns of educational videos). Majority of the
research in this domain has been based on ASSISTments data(Selent et al.,
2016), that records a series of learners solving tests in an ITS platform.
These datasets are heavily biased towards mathematics knowledge as the
ASSISTments tool was used for mathematics education from the beginning(Fyfe,
2016b; McGuire et al., 2017).

A few publicly available datasets to solve knowledge tracing problem exist.
They include mathematics
ASSISTments111http://www.assistmentstestbed.org/(Hurst and Cordes, 2018;
Walkington et al., 2019), problem solving interactions(Choi et al., 2020), or
multiple choice questions222https://www.microsoft.com/en-
us/research/event/diagnostic-questions-neurips2020/(Wang et al., 2020, 2021)),
all of which are noteworthy although they do not focus on implicit feedback
and engagement. MOOCCube is a very recently released dataset that contains a
spectrum of different statistics relating to learner-MOOC interactions
including implicit and explicit test taking activity(Yu et al., 2020).
Although this dataset may contain data that can be used to predict learner
engagement with implicit feedback, the dataset has only been used in
prerequisite detection task which is very different. However, our prior
experiments have consistently demonstrated that engagement prediction via
online learning(Bulathwela et al., 2020e) can be achieved with a dataset that
is similar to PEEK in structure.

Although previous work based on popular video repositories such as edX(Guo et
al., 2014), Khan Academy(MacHardy and Pardos, 2015), VideoLectures(Bulathwela
et al., 2020e, d) and YouTube(Wu et al., 2018; Covington et al., 2016) has
evidenced the existence of datasets that include implicit engagement signals,
these datasets are never made public due to their proprietary nature. The
scarcity of large scale publicly available datasets for predicting learner
engagement with educational videos constrain the growth of the field. In this
context, the Personalised Educational Engagement (_PEEK_) dataset, is the
first and largest learner video engagement dataset that will be publicly
released with humanly-interpretable Wikipedia concepts and the concept
coverage associated with the video lecture fragments.

##  3\. PEEK Dataset

In this section, we describe how the Personalised Educational Engagement
linked to Knowledge topics (PEEK) dataset is constructed. Figure 2 outlines
the overall process of creating the PEEK dataset.

###  3.1. Data Source

PEEK dataset is constructed using video metadata and learner activity data
extracted from VideoLectures.Net333www.videolectures.net (VLN), a repository
of scientific and educational video lectures. VLN repository records research
talks and presentations from numerous academic venues accompanied by the
lecture slides associated with the video. As the talks are recorded at peer-
reviewed conferences and prestigious research venues, the lectures are
reviewed and material is controlled for correctness of knowledge.

Figure 1 depicts the different components of the VLN user interface where a
user engages with an educational video. Every lecture has a title, event
details (e.g. conference venue, date, location etc.) and lecture meta data
associated with it. Although most lectures consist of one video, some video
lectures may contain more than one video as shown in Figure 1.

Figure 1. Screen layout of VideoLecture.Net website from which the data for
PEEK dataset was sourced. Every lecture can have one or more videos and is
associated with metadata. The lecture transcripts are also available for
processing.

Figure 2. The video data and the learner interaction logs from VLN repository
are processed separately to create the Wikipedia-based KCs and also the
discrete engagement signals that are published in PEEK dataset.

###  3.2. Fragmenting Videos

First, the videos in VLN repository are transcribed to its native language
using the _TransLectures_ project444www.translectures.eu. Then the non-English
lecture videos are translated to English as we will use English Wikipedia for
entity linking.

As described in Section 3.1, a lecture in VLN repository can contain one or
more videos. To model user interactions in a much more granular level we
further break each video into smaller parts. Once the
transcription/translation is complete, we partition the transcript of each
video into multiple _fragments_ where each fragment covers approximately 5
minutes of lecture time. Having 5 minute fragments allow us to break the
contents of a video into a more granular level while making sure that there is
sufficient amounts of information in each video fragment.

###  3.3. Wikification of Transcripts

In order to identify the Knowledge Components (KCs) that are contained in
different video fragments, we use Wikification(Brank et al., 2017). This
allows annotating learning materials with humanly interpretable KCs (Wikipedia
concepts) at scale with minimum human-expert intervention. This setup will
make sure that recommendation strategies build on this dataset will be
technologically feasible for web-scale e-learning systems. Previous works have
demonstrated the value of using Wikification in this task(Bulathwela et al.,
2020e). We restrict the dataset to English Wikipedia due to its richness in
comparison to other languages.

Figure 3. Visual representation of the different data items available in the
PEEK dataset. Each video is broken into multiple, non-overlapping 5 minute
fragments that are linked with ranked Wikipedia-based KCs. The watched parts
of the video (in red) are used to create discrete engagement labels.

###  3.4. Knowledge Component Ranking

As per (Brank et al., 2017), Wikification produces two statistical values per
annotated KC, namely, _PageRank_ and _Cosine Similarity_ scores.

_PageRank score_ is calculated by constructing a semantic graph where semantic
relatedness ($SR(c,c^{\prime})$) between each Wikipedia concept pair $c$ and
$c^{\prime}$ in the graph are calculated using equation 1.

(1) $SR(c,c^{\prime})=\frac{\log(max(|L_{c}|,|L_{c^{\prime}}|)-\log(|L_{c}\cap
L_{c^{\prime}}|)}{\log|W|-log(min(|L_{c}|,|L_{c^{\prime}}|)}$

where $L_{c}$ represents the set of Wiki concepts with inwards links to
Wikipedia concept $c$, $|\cdot|$ represents the cardinality of the set and $W$
represents the set of all Wikipedia topics. This semantic relatedness graph is
used for computing PageRank scores. PageRank algorithm(Brin and Page, 1998)
leads to heavily connected Wikipedia topics (i.e. more semantically related)
within the lecture to get a higher score.

The _Cosine Similarity score_ is used as a proxy for topic coverage within the
lecture fragment(Bulathwela et al., 2020e). This score $cos(s_{tr},c)$ between
the _Term Frequency-Inverse Document Frequency (TF-IDF)_ representations of
the lecture transcript $s_{tr}$ and the Wikipedia page $c$ is calculated based
on equation 2:

(2)
$cos(s_{tr},c)=\frac{\texttt{TFIDF}(s_{tr})\cdot\texttt{TFIDF}(c)}{\|\texttt{TFIDF}(s_{tr})\|\times\|\texttt{TFIDF}(c)\|}$

where $\texttt{TFIDF}(s)$ returns the TF-IDF vector of the string $s$ while
$||\cdot||$ represents the norm of the TF-IDF vector.

The authors of (Brank et al., 2017) comment that a linearly weighted sum
between the PageRank and Cosine score can be used to rank the importance of
Wikipedia concepts as per equation 3.

(3) $\texttt{Rank}(c)=\alpha\cdot PageRank(c)+(1-\alpha)\cdot
cos(s_{tr},c),\text{where }\alpha\in[0,1]$

We experimented with different values for $\alpha$ in equation 3 and
empirically validated suitable linear combinations of weights for PageRank
score and Cosine Similarity score. We observed that a weight of 0.8 on
PageRank and 0.2 on Cosine similarity leads to the most suitable ranking of
KCs(Bulathwela et al., [n.d.]). We use Rank to identify the five top-ranked
KCs for each lecture fragment. The cosine similarity score as per equation 2
is included in the dataset as a proxy for coverage of that KC in the lecture
fragment. We restrict the number of KCs to 5 as larger numbers (e.g. 10 KCs)
have shown to degrade performance of learner models built with similar
datasets(Bulathwela et al., 2020e). Figure 4(ii) provides a word cloud of the
most dominant KCs in PEEK dataset. It is evident that the majority of KCs
(Wikipedia topics) associated to the lecture fragments in this dataset are
related to artificial intelligence and machine learning.

###  3.5. Anonymity

We restrict the final dataset to lectures that have been viewed by at least 5
unique users to preserve k-anonymity(Craswell et al., 2020) of users. Also, we
report the timestamp of user view events in relation to the earliest event
found in the dataset obfuscating the actual timestamp. That is we report the
smallest timestamp in the dataset $t_{0}$ as 0s and any timestamp $t_{i}$
after that as $t_{i}-t_{0}$. This allows us to publish the true order and the
real differences of duration between events without revealing the actual
timestamps. Additionally, the lecture metadata such as title and authors etc.
are not published to preserve the anonymity of the authors/lecturers. The
motivation behind this decision is to avoid authors of the video lectures
having unanticipated effects on their reputation by associating implicit
learner engagement values to their content.

###  3.6. Labels

The user interface of VLN website also records the video watching behaviours
of its users (Interaction logs in Figure 2).

The target label for learner engagement is a discrete variable based on _video
watch time_ which has been used as a proxy for video engagement in both non-
educational(Covington et al., 2016; Wu et al., 2018) and educational(Guo et
al., 2014; Bulathwela et al., 2020e) contexts. Normalised learner watch time
$\overline{e}^{t}_{\ell,r}$ of learner $\ell$ with video fragment resource
$r_{i}$ at time point $t$ is calculated as per equation 4.

(4)
$\centering\overline{e}^{t}_{\ell,r_{i}}=W(\ell,r_{i})/D(r_{i}),\@add@centering$

where $\overline{e}^{t}_{\ell,r_{i}}\in\\{0,1\\}$, $W(\cdot)$ is a function
that returns the _watch time_ of learner $\ell$ for resource $r_{i}$ and
$D(\cdot)$ is a function that returns the duration of lecture fragment
$r_{i}$. The ultimate label $e^{t}_{\ell,r_{i}}$ is derived by discretising
$\overline{e}^{t}_{\ell,r_{i}}$ where $e^{t}_{\ell,r_{i}}=1$ when
$\overline{e}^{t}_{\ell,r_{i}}\geq.75$ and $e^{t}_{\ell,r_{i}}=0$ otherwise.
The discretisation rule is motivated by the hypothesis that a learner should
watch approximately 4 minutes of an educational video fragment that is
approximately 5 minutes (duration of a video fragment that includes 5000
characters from the video transcript) in order to acquire knowledge from
it(Bulathwela et al., 2020e).

###  3.7. Final Dataset

The final PEEK dataset consists of 290,535 interaction events from 20,019
distinct users with at least 5 events. These learner engage with 10,233 unique
lecture videos that are partitioned into 39,113 fragments (3.82 fragments per
video). The learner population in the dataset is divided into _Training_
(14,050 learners) and _Test_ (5,969 learners) datasets based on a 70:30 split.
The label distribution in the dataset is also relatively balanced with only
56% of the labels being positive. As shown in Figure 4 (i), the majority of
learners in the dataset have a relatively small number of events (under 80)
making this dataset an excellent test bed for personalisation models designed
to work in data scarce environments. VLN repository mainly publishes videos
relating to Computer Science and Machine Learning leading to a learner
audience who visit to learn about these subjects. This fact is confirmed by
Figure 4 where it shows that the dataset is dominated by events with AI and ML
related KCs.

###  3.8. Structure of the PEEK Dataset

The final dataset consists of 3 files.

  1. (1)

train.csv, used for hyperparameter validation and training parameters.

  2. (2)

test.csv, used as the held-out test set.

  3. (3)

id_url_mapping.csv, which contains the mapping between KC IDs and Wikipedia
page URL.

train.csv and test.csv files contain the actual learner session data where
their interaction with lecture fragments are recorded. The two files contains
70% and 30% of the learners respectively. Both files contain 15 columns that
are described in Table 1. As the name suggests id_url_mapping.csv file
contains a mapping between the KC ID in PEEK dataset and the URL of the
Wikipedia page of the concept associated with that KC.

Figure 4. Characteristics of the PEEK dataset: (i) number of learners in the training/test dataset based on the number of events in the session for individual learner and (ii) wordclouds depicting the most frequently detected Wikipedia-based knowledge components. Table 1. Detailed descriptions of the different columns of the train.csv and test.csv files included in the PEEK Dataset. Column Number | Description | Details
---|---|---
1 | Video Lecture ID | An integer ID associated with an individual video lecture
2 | Video ID | An integer ID associated to every video belonging to the same Video Lecture ID (e.g. $1\dots v$ if
|  | the lecture with $v$ videos)
3 | Part ID | An integer ID associated with each video fragment (e.g $1\dots f$ for a video with $f$ fragments)
4 | Timestamp | Timestamp (to the nearest second) when the play event was initiated.
5 | user ID | An integer ID associated with each unique learner in the dataset PEEK dataset (IDs in
|  |  train.csv and test.csv files are mutually exclusive).
6,8,10,12,14 | KC IDs | An integer ID associated with each unique Knowledge Component in the dataset. This ID can
|  | be linked to the human readable Wikipedia concept names found in url_id_mapping.csv file
7,9,11,13,15 | Topic Coverage | Proxy for coverage of the relevant KC in the fragment of interest. KC coverage is the cosine
|  | similarity as per equation 2.
16 | Label | The binary label $e^{t}_{\ell,r_{i}}$, 1 if the learner watched $\geq$ .75 of the video fragment, 0 otherwise.

##  4\. Main Prediction Task and Models

We benchmark the predictive performance of this dataset with respect to
learner engagement prediction by proposing a set of baseline models and
measuring their performance with the PEEK dataset.

###  4.1. Data

Unlike previous studies, which relied on small, expert-curated collections of
open educational materials, our dataset is built using in-the-wild watch
patterns of multi-lingual open educational materials available online.

We construct two distinct datasets for our experiments. (i) Active 20 is a
smaller dataset that consists of 8,207 interaction events from the 20 most
active users (410.35 events per learner) in the PEEK dataset. This dataset
contains a substantially small number of users who have relatively longer
sessions. The smaller dataset will allow us to understand how the proposed
learner models will behave with mature learners that have more interaction
information in the system. On the contrary, (ii) PEEK is the full dataset that
consists of 290,535 interaction events from 20,019 distinct users with at
least 5 events. However, the majority of users in this dataset have relatively
smaller sessions (average of 14.51 events per learner).

###  4.2. Prediction Task and Evaluation Metrics

As the dataset is already divided into train and test splits, all our
experiments use a hold-out validation (train-test split) approach where
hyperparameters and parameters are learned on 70% of the learners and the
model is evaluated on the remaining 30% with the selected hyperparameter
combination.

A sequential experimental design is employed, where
$\hat{e}^{t}_{\ell,r_{i}}$, engagement with fragment $t$ is predicted using
fragments $1$ to $t-1$. Since engagement is binary, predictions for each
fragment can be assembled into a confusion matrix, from which we compute well-
known binary classification metrics such as precision, recall and F1-measure.
We focus on these measures as we are more interested in predictive lecture
fragments that the learners are likely to engage with.

We average these metrics per learner and weight each learner according to
their amount of activity in the system. We use F1-measure for model comparison
as we are interested in improving both precision and recall.

The primary objective of the experiments is to evaluate how the different
attributes of the dataset are useful in modelling engagement of learners.
Towards this goal, we run two primary experiments, 1) investigate how
different recommendation models perform with predicting engagement, and 2) how
the number of KCs impact the prediction model. The benchmark algorithms used
for the experiments are outlined in Section 4.3. The results of the
experiments are outlined in Section 5.

###  4.3. Benchmark Models

PEEK is the first of its kind, a dataset that records in-the-wild engagement
of informal learners with video lecture fragments. Due to the novelty of this
dataset, we struggle to find already published baselines, except for the
TrueLearn family of algorithms(Bulathwela et al., 2020e). For the sake of
comparing its predictive performance, we also propose a set of baselines that
are based on content-based and collaborative filtering.

#### Content-based Similarity

Content-based filtering can measure the similarity between two items. We
compute a similarity value, $sim(r^{t-1}_{\ell,r_{i}},r^{t}_{\ell,r_{i}})$
between two consecutive lecture fragments $r^{t-1}_{\ell,r_{i}}$ and
$r^{t}_{\ell,r_{i}}$ in the learner $\ell$’s session. We use this similarity
value to make an engagement prediction $\hat{e}^{t}_{\ell,r_{i}}$ based on
equation 5.

(5) $\centering\hat{e}^{t}_{\ell,r_{i}}=\left\\{\begin{array}[]{ c l
}1&\text{\ if\ }sim(r^{t-1}_{\ell,r_{i}},r^{t}_{\ell,r_{i}})\geq threshold\\\
0&\text{otherwise}\end{array}\right.\@add@centering$

In this case, we investigate two similarity measures, namely 1) _Cosine_ , 2)
_Concept-based Jaccard_ and _User-based Jaccard_. When computing cosine
similarity, we represent each video fragment using the bag of concepts
representation where the concepts are the super set of Wikipedia concepts
mentioned in the dataset. The values in this sparse vector are the cosine
similarities between respective Wikipedia concept and the lecture fragment
transcript as per equation 2.

An alternative approach to finding concept-wise similarity is Jaccard
similarity. Concept-based Jaccard similarity
$\texttt{Jaccard}_{\mathcal{C}}(r^{t-1}_{\ell,r_{i}},r^{t}_{\ell,r_{i}})$,
between lecture fragments $r^{t-1}_{\ell,r_{i}}$ and $r^{t}_{\ell,r_{i}}$ is
computed based on equation 6.

(6)
$\centering\texttt{Jaccard}_{\mathcal{C}}(r^{t-1}_{\ell,r_{i}},r^{t}_{\ell,r_{i}})=\frac{\mathcal{C}(r^{t-1}_{\ell,r_{i}})\cap\mathcal{C}(r^{t}_{\ell,r_{i}})}{\mathcal{C}(r^{t-1}_{\ell,r_{i}})\cup\mathcal{C}(r^{t}_{\ell,r_{i}})}\@add@centering$

where $\mathcal{C}(\cdot)$ is a function that returns the set of Wikipedia
concepts in resource $r_{i}$

Similarly, one can also measure the similarity between two lecture fragments
based on how many learners interact with both the lecture fragments. The user
interactions in the training dataset is used exclusively to learn the
similarity matrix in order to avoid data leakage. In this approach, we can
calculate the user-wise jaccard similarity
$\texttt{Jaccard}_{\mathcal{U}}(r^{t-1}_{\ell,r_{i}},r^{t}_{\ell,r_{i}})$, as
per equation 7.

(7)
$\centering\texttt{Jaccard}_{\mathcal{U}}(r^{t-1}_{\ell,r_{i}},r^{t}_{\ell,r_{i}})=\frac{\mathcal{U}(r^{t-1}_{\ell,r_{i}})\cap\mathcal{U}(r^{t}_{\ell,r_{i}})}{\mathcal{U}(r^{t-1}_{\ell,r_{i}})\cup\mathcal{U}(r^{t}_{\ell,r_{i}})}\@add@centering$

where $\mathcal{U}(\cdot)$ is a function that returns the set of learners that
interacted with resource $r_{i}$

#### Knowledge Tracing (KT)

KT builds a learner representation of knowledge of the learner(Yudelson et
al., 2013). This learning model is then used in predicting engagement of
learner $\ell$ with lecture fragment resource $r_{i}$ at time $t$. As the PEEK
dataset has a temporal dimension, we reformulate the KT algorithm into an
online learning graphical model inspired by the reformulation in(Bishop et
al., 2015). The skill variables in the KT model are Bernoulli variables
($\theta^{t}_{\ell,c}\sim\texttt{Bernoulli}(\pi^{t}_{\ell,c})$), assuming that
a learner $\ell$ would have either mastered a skill/ concept $c$ or not
(represented by probability $\pi^{t}_{\ell,c}$). Skills are initialised
($\theta^{0}_{\ell,c}$) using a $\texttt{Bernoulli}(.0)$ prior, assuming that
the latent skill is not mastered in the beginning. A noise factor similar to
what is found in the conventional KT model(Corbett and Anderson, 1994) is
added to this model and is tuned using a grid search.

#### TrueLearn Novel Model

Similar to KT model, TrueLearn model is also an online, graphical model that
develops a learner model of the learner. This model is inspired by the
TrueSkill model(Herbrich et al., 2007) which is an evolution of IRT(Rasch,
1960). Contrary to the KT model, TrueLearn model models skills as Gaussian
variables ($\theta^{t}_{\ell,c}\sim\mathcal{N}(\mu,\sigma^{2})$. In addition
to modelling knowledge, TrueLearn Novel model also models novelty of content
which is a key aspect of educational recommendation(Bulathwela et al., 2020e).
Enforcing the same assumptions of KT model, the TrueLearn Novel skill
parameters are initialised using a $\mathcal{N}(.0,\sigma_{0}^{2})$ prior
where $\sigma_{0}^{2}$, initial variance is a hyperparameter tuned using a
grid search.

##  5\. Prediction Task Performance and Discussion

The predictive performance of multiple benchmark model (outlined in Section
4.3) with the PEEK dataset is evaluated. The hyperparameters of each model
(including number of KCs) are tuned using a grid search. The model performance
with the best hyperparameter combinations is presented in Table 2.

Table 2. Predictive performance of suitable  Dataset | Most Active 20 Users | PEEK Dataset
---|---|---
Algorithm | Precision | Recall | F1-Score | Precision | Recall | F1-Score
Cosine Similarity | 67.799 | 87.194 | 73.635 | 57.859 | 58.451 | 54.056
$\texttt{Jaccard}_{\mathcal{C}}$ Similarity | 67.157 | 80.901 | 70.818 | 57.810 | 60.362 | 55.026
$\texttt{Jaccard}_{\mathcal{U}}$ Similarity | 46.499 | 11.606 | 17.761 | 57.854 | 72.755 | 61.219
Knowledge Tracing | 65.050 | 65.828 | 62.077 | 53.254 | 28.564 | 34.513
TrueLearn Novel | 78.790 | 89.390 | 83.090 | 58.290 | 79.240 | 64.710

Good predictive power of the similarity based approaches validate the
expressiveness of the content representations based on Wikipedia concepts and
cosine similarity. The poor performance of $Jaccard_{\mathcal{U}}$ Similarity
model with the top 20 user dataset is expected as this model relies on
learning similarities based on training data. Although 20 user dataset
contains active users with more events per session, there is less data in
total due to the small number of users which leads to learning a mostly
incomplete similarity matrix between lecture fragments. It is seen that this
condition changes with the full PEEK dataset where more data is available. The
results in table 2 shows that TrueLearn Novel model performs best among the
benchmarks investigated. This result further reinforces the superiority of
TrueLearn Novel model investigated with a similar dataset(Bulathwela et al.,
2020e).

###  5.1. Number of Knowledge Components

The impact of the number of KCs for each model is also investigated and the
results are reported in Figure 5.

Figure 5. Predictive performance of PEEK dataset in terms of Precision (left),
Recall (middle) and F1-Score (right) for the benchmark models when varying
numbers of Knowledge Components (KCs) are used as the content representation.

The plots indicate that the similarity based approaches tend to perform better
with more KCs in the representation. This is natural as Similarity based
approaches need more information from the exact pair of contents that are
being compared in order to make a fine grained prediction. On the contrary,
Knowledge Tracing and TrueLearn novel Models that develop a learner model
perform better with less number of KCs. This observation is also expected as
the learner model has the capacity to store information about various
different KCs that the learner encountered over the past. This allows the
learner model to make a good prediction with little information from the
current resource. In the TrueLearn model, the effect of adding extra KCs to
the representation makes a very small difference in performance.

###  5.2. Discussion

Knowledge (skill mastery to be more specific) is usually measured from
explicit test performance(Corbett and Anderson, 1994; Piech et al., 2015;
Nakagawa et al., 2019). Contrary to narrowly scoped learning scenarios common
to ITS, seeking explicit feedback is comparatively less realistic in lifelong
learning settings where informal learners attempt to acquire knowledge without
necessarily having the need to go through rigorous testing. Forcing learners
into aggressive testing can hinder their learning experience. Additionally,
technical challenges too exist in areas such as scalable test/ question
generation and automatic test marking. However,(Bulathwela et al., 2020e) has
demonstrated that modelling knowledge from educational watch patterns is a
promising direction towards predicting future engagement. The opportunity that
PEEK dataset brings is the capability to advance this research avenue by
making the first large dataset publicly available. This dataset will allow
researchers to improve knowledge representation and also include other crucial
factors such as learner interest(Bulathwela et al., 2020d) in the spirit of
building integrative educational recommenders using implicit feedback. As
Figure 4(i) shows, PEEK dataset also contains a large number of learners with
limited activity making this dataset an excellent candidate to evaluate low
resource, data efficient personalisation algorithms.

PEEK dataset also acts as the bridge between multiple rich data sources. The
Wikipedia concepts used in representing the KCs also enables connecting this
dataset to the variety of auxiliary information available around Wikipedia,
the world’s largest encyclopedia. Some examples of possibilities are the
ability to leveraging additional data structures such as Wikipedia page
contents, the hyperlink graph and the category tree. Other enriched
derivatives of Wikipedia such as semantic relatedness(Ponza et al., 2020) and
knowledge bases(Auer et al., 2007; Wikimedia, 2018) are also directly fused
with this dataset to build more powerful content representations. In addition,
the Wikipedia taxonomy also provides humanly intuitive representations
(grounded on Wikipedia concepts) that enhances interpretability. Figure 4 (ii)
is a prime example of how humanly intuitive KCs improves the expressiveness of
content representations. A humanly intuitive representation also paves way to
much needed interpretable learner models that are essential to triggering
meta-cognition within learners(Bull and Kay, 2008).

###  5.3. Relevance to Online Learning

PEEK dataset consists of a collection of learners making engagement choices
across educational materials over time. This dataset clearly captures the
temporal dynamics of the learners whose knowledge state and preferences change
over time. The superiority of TrueLearn Novel in Table 2 also gives a strong
indication on how a state-aware learner model that changes its learner
representation over time, best captures the engagement dynamics of the dataset
in comparison to naïve similarity-based approaches. This result reaffirms the
relevance and usefulness of this dataset to invent online learning models that
can identify personal and temporal dynamics of users.

###  5.4. Supported Tasks

This section introduces the reader to the tasks that the PEEK dataset could be
used for. The main application area of the PEEK dataset is engagement
prediction with video lecture fragments in an web-based learning setting as
demonstrated in Section 4. However, this task can be further extended by
fusing the engagement labels from all the fragments of a video to carry out
video recommendation which is more common(Covington et al., 2016) (contrary to
video fragment recommendation).

Moving away from engagement prediction, PEEK dataset can also be used to
understand the structure of knowledge and learning pathways through it.
Deducing the structure of knowledge using the co-occurrence patterns of KCs
within the video lectures provide opportunities to understand inter-topic
relationships and how knowledge is structured. Work in this direction can be
used in identifying related materials and accounting for novelty in
educational recommendation(Bulathwela et al., 2020e). The dataset can also be
used to identify different clusters/groups of learners and learning resources
which will allow understanding the education landscape better. Studying the
evolution of KCs within learner journeys over time can also unlock
opportunities to understand prerequisites(Yu et al., 2020).

##  6\. Conclusions

This work releases _PEEK_ , the first and largest publicly-available, humanly-
intuitive dataset of learner interaction with educational videos opening up
various avenues to the research community to push the frontiers of this line
of research. It publishes learner interactions of over 20,000 learners with
fragments of lectures over time. The predictive performance of several
benchmark models is evaluated with this dataset. The power of this dataset is
already portrayed by the promising experimental results presented.

As future directions, we see the importance of modelling semantic relatedness
between KCs(Ponza et al., 2020) that are likely to boost performance of
learner models such as Knowledge Tracing and TrueLearn. Exploiting other side
information from other Wikipedia such as the category tree is also an
interesting direction to explore. The potential of detecting other user
signals such as learner interest from PEEK dataset can also lead to insightful
findings. PEEK dataset also allows experimenting with temporal dynamics such
as interest decay(Piao and Breslin, 2016).

###### Acknowledgements.

This work is partially supported by the European Commission funded project
”Humane AI: Toward AI Systems That Augment and Empower Humans by Understanding
Us, our Society and the World Around Us” (grant 820437) and the EPSRC
Fellowship titled ”Task Based Information Retrieval” (grant EP/P024289/1).

## References

  * (1)
  * Auer et al. (2007) Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007\.  Dbpedia: A nucleus for a web of open data.  In _The semantic web_. Springer, 722–735.
  * Barria Pineda and Brusilovsky (2019) Jordan Barria Pineda and Peter Brusilovsky. 2019.  Making educational recommendations transparent through a fine-grained open learner model. In _Proceedings of Workshop on Intelligent User Interfaces for Algorithmic Transparency in Emerging Technologies at the 24th ACM Conference on Intelligent User Interfaces, IUI 2019, Los Angeles, USA, March 20, 2019_ , Vol. 2327.
  * Bauman and Tuzhilin (2018) Konstantin Bauman and Alexander Tuzhilin. 2018.  Recommending remedial learning materials to students by filling their knowledge gaps.  _MIS Quarterly_ 42, 1 (2018), 313–332.
  * Bishop et al. (2015) Christopher Bishop, John Winn, and Tom Diethe. 2015.  _Model-Based Machine Learning_.  Early access version (http://www.mbmlbook.com/).  Accessed: 2019-05-23.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003\.  Latent Dirichlet Allocation.  _Journal of Machine Learning Research_ 3 (2003).
  * Brank et al. (2017) Janez Brank, Gregor Leban, and Marko Grobelnik. 2017\.  Annotating Documents with Relevant Wikipedia Concepts. In _Proc. of Slovenian KDD Conf. on Data Mining and Data Warehouses (SiKDD)_ (Ljubljana, Slovenia).
  * Brin and Page (1998) Sergey Brin and Lawrence Page. 1998.  The anatomy of a large-scale hypertextual Web search engine. In _Proc. of Int. Conf. on World Wide Web_.
  * Bulathwela et al. (2020a) Sahan Bulathwela, Stefan Kreitmayer, and María Pérez-Ortiz. 2020a.  What’s in It for Me? Augmenting Recommended Learning Resources with Navigable Annotations. In _Proc. of Int. Conf. on Intelligent User Interfaces Companion_ _(IUI ’20)_. 114–115.
  * Bulathwela et al. (2020b) Sahan Bulathwela, María Pérez-Ortiz, Aldo Lipani, Emine Yilmaz, and John Shawe-Taylor. 2020b.  Predicting Engagement in Video Lectures. In _Proc. of Int. Conf. on Educational Data Mining_ _(EDM ’20)_.  https://arxiv.org/pdf/2006.00592.pdf
  * Bulathwela et al. (2020c) Sahan Bulathwela, María Pérez-Ortiz, Rishabh Mehrotra, Davor Orlic, Colin de la Higuera, John Shawe-Taylor, and Emine Yilmaz. 2020c.  SUM’20: State-Based User Modelling. In _Proceedings of the 13th International Conference on Web Search and Data Mining_ _(WSDM ’20)_. Association for Computing Machinery, New York, NY, USA.  https://doi.org/10.1145/3336191.3371883
  * Bulathwela et al. (2021) Sahan Bulathwela, María Pérez-Ortiz, Rishabh Mehrotra, Davor Orlic, Colin de la Higuera, John Shawe-Taylor, and Emine Yilmaz. 2021.  Report on the WSDM 2020 Workshop on State-Based User Modelling (SUM’20).  _SIGIR Forum_ 54, 1, Article 5 (Feb. 2021).  https://doi.org/10.1145/3451964.3451969
  * Bulathwela et al. ([n.d.]) Sahan Bulathwela, María Pérez-Ortiz, E. S. V. Ranawaka, R. I. P. B. B. Siriwardana, G. A. K. Y. Ganepola, Emine Yilmaz, and John Shawe-Taylor. [n.d.].  D1.6 – Report on Selected Models and Content Representations.  https://www.x5gon.org/wp-content/uploads/2021/01/D1.6-1.pdf.  Accessed in: 2021-04-02.
  * Bulathwela et al. (2020d) Sahan Bulathwela, María Pérez-Ortiz, Emine Yilmaz, and John Shawe-Taylor. 2020d.  Towards an Integrative Educational Recommender for Lifelong Learners. In _AAAI Conference on Artificial Intelligence_.
  * Bulathwela et al. (2020e) Sahan Bulathwela, María Pérez-Ortiz, Emine Yilmaz, and John Shawe-Taylor. 2020e.  TrueLearn: A Family of Bayesian Algorithms to Match Lifelong Learners to Open Educational Resources. In _AAAI Conference on Artificial Intelligence_.
  * Bull (2020) Susan Bull. 2020\.  There are open learner models about!  _IEEE Transactions on Learning Technologies_ 13, 2 (2020), 425–448.
  * Bull and Kay (2008) Susan Bull and Judy Kay. 2008.  Metacognition and open learner models. In _The 3rd workshop on meta-cognition and self-regulated learning in educational technologies, at ITS2008_. 7–20.
  * Chen et al. (2018) Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. 2018.  Temporally grounding natural sentence in video. In _Proceedings of the 2018 conference on empirical methods in natural language processing_. 162–171.
  * Choi et al. (2020) Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Seoyon Park, Seewoo Lee, Jineon Baek, Chan Bae, Byungsoo Kim, and Jaewe Heo. 2020\.  Ednet: A large-scale hierarchical dataset in education. In _International Conference on Artificial Intelligence in Education_. Springer, 69–73.
  * Corbett and Anderson (1994) Albert T. Corbett and John R. Anderson. 1994.  Knowledge tracing: Modeling the acquisition of procedural knowledge.  _User Modeling and User-Adapted Interaction_ 4, 4 (1994).
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016\.  Deep Neural Networks for YouTube Recommendations. In _Proc. of ACM Conf. on Recommender Systems_.
  * Craswell et al. (2020) Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Billerbeck. 2020.  ORCAS: 20 Million Clicked Query-Document Pairs for Analyzing Search. In _Proceedings of the 29th ACM International Conference on Information and Knowledge Management_ (Virtual Event, Ireland) _(CIKM ’20)_. Association for Computing Machinery, New York, NY, USA, 2983–2989.  https://doi.org/10.1145/3340531.3412779
  * Dumais (2004) Susan T Dumais. 2004\.  Latent semantic analysis.  _Annual review of information science and technology_ 38, 1 (2004), 188–230.
  * Egozi et al. (2011) Ofer Egozi, Shaul Markovitch, and Evgeniy Gabrilovich. 2011\.  Concept-based information retrieval using explicit semantic analysis.  _ACM Transactions on Information Systems (TOIS)_ 29, 2 (2011), 1–34.
  * Ferragina and Scaiella (2010) Paolo Ferragina and Ugo Scaiella. 2010.  TAGME: On-the-Fly Annotation of Short Text Fragments (by Wikipedia Entities). In _Proc. of ACM Int. Conf. on Information and Knowledge Management_ _(CIKM ’10)_.
  * Fyfe (2016a) Emily R Fyfe. 2016a.  Providing feedback on computer-based algebra homework in middle-school classrooms.  _Computers in Human Behavior_ 63 (2016), 568–574.
  * Fyfe (2016b) Emily R. Fyfe. 2016b.  Providing Feedback on Computer-Based Algebra Homework in Middle-School Classrooms.  _Comput. Hum. Behav._ 63, C (Oct. 2016), 568–574.  https://doi.org/10.1016/j.chb.2016.05.082
  * Gabrilovich et al. (2007) Evgeniy Gabrilovich, Shaul Markovitch, et al. 2007\.  Computing semantic relatedness using Wikipedia-based explicit semantic analysis.. In _IJcAI_ , Vol. 7. 1606–1611.
  * Google (2021) Google. 2021.  YouTube Help: Video chapters.  https://support.google.com/youtube/answer/9884579?hl=en-GB
  * Guo et al. (2014) Philip J. Guo, Juho Kim, and Rob Rubin. 2014.  How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. In _Proc. of the First ACM Conf. on Learning @ Scale_.
  * Herbrich et al. (2007) Ralf Herbrich, Tom Minka, and Thore Graepel. 2007.  TrueSkill(TM): A Bayesian Skill Rating System. In _Advances in Neural Information Processing Systems 20_. 569–576.  https://www.microsoft.com/en-us/research/publication/trueskilltm-a-bayesian-skill-rating-system/
  * Hofmann (2013) Thomas Hofmann. 2013\.  Probabilistic latent semantic analysis.  _arXiv preprint arXiv:1301.6705_ (2013).
  * Hurst and Cordes (2018) Michelle Hurst and Sara Cordes. 2018.  Labeling Common and Uncommon Fractions Across Notation and Education. In _Proceedings of the 40th Annual Meeting of the Cognitive Science Society, CogSci 2018, Madison, WI, USA, July 25-28, 2018_. cognitivesciencesociety.org.  https://mindmodeling.org/cogsci2018/papers/0354/index.html
  * Jiang et al. (2019) Weijie Jiang, Zachary A. Pardos, and Qiang Wei. 2019.  Goal-based Course Recommendation. In _Proceedings of International Conference on Learning Analytics & Knowledge_.
  * Katz et al. (2011) Gilad Katz, Nir Ofek, Bracha Shapira, Lior Rokach, and Guy Shani. 2011.  Using Wikipedia to boost collaborative filtering techniques. In _Proceedings of the fifth ACM conference on Recommender systems_. 285–288.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020\.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _arXiv preprint arXiv:2005.11401_ (2020).
  * Liang (2019) Shangsong Liang. 2019\.  Collaborative, dynamic and diversified user profiling. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 4269–4276.
  * MacHardy and Pardos (2015) Zachary MacHardy and Zachary A. Pardos. 2015.  Evaluating The Relevance of Educational Videos using BKT and Big Data. In _Proc. of Int. Conf. on Educational Data Mining_.
  * Mahapatra et al. (2018) Debabrata Mahapatra, Ragunathan Mariappan, Vaibhav Rajan, Kuldeep Yadav, Seby A., and Sudeshna Roy. 2018.  VideoKen: Automatic Video Summarization and Course Curation to Support Learning. In _Companion Proceedings of the The Web Conference 2018_ (Lyon, France) _(WWW ’18)_. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 239–242.  https://doi.org/10.1145/3184558.3186988
  * McGuire et al. (2017) Patrick McGuire, Shihfen Tu, Mary Ellin Logue, Craig A Mason, and Korinn Ostrow. 2017.  Counterintuitive effects of online feedback in middle school math: results from a randomized controlled trial in ASSISTments.  _Educational Media International_ 54, 3 (2017), 231–244.
  * Nakagawa et al. (2019) Hiromi Nakagawa, Yusuke Iwasawa, and Yutaka Matsuo. 2019\.  Graph-Based Knowledge Tracing: Modeling Student Proficiency Using Graph Neural Network. In _IEEE/WIC/ACM International Conference on Web Intelligence_ _(WI ’19)_.
  * Pardos et al. (2014) Zachary A Pardos, Ryan SJD Baker, Maria OCZ San Pedro, Sujith M Gowda, and Supreeth M Gowda. 2014\.  Affective States and State Tests: Investigating How Affect and Engagement during the School Year Predict End-of-Year Learning Outcomes.  _Journal of Learning Analytics_ 1, 1 (2014).
  * Pavlik et al. (2009) Philip I. Pavlik, Hao Cen, and Kenneth R. Koedinger. 2009\.  Performance Factors Analysis –A New Alternative to Knowledge Tracing. In _Proc. of Int. Conf. on Artificial Intelligence in Education_.
  * Pelánek et al. (2017) Radek Pelánek, Jan Papoušek, Jiří Řihák, Vít Stanislav, and Juraj Nižnan. 2017.  Elo-based learner modeling for the adaptive practice of facts.  _User Modeling and User-Adapted Interaction_ 27, 1 (01 Mar 2017), 89–118.
  * Pérez-Ortiz et al. (2021) María Pérez-Ortiz, Claire Dormann, Yvonne Rogers, Sahan Bulathwela, Stefan Kreitmayer, Emine Yilmaz, Richard Noss, and John Shawe-Taylor. 2021.  X5Learn: A Personalised Learning Companion at the Intersection of AI and HCI _(IUI ’21)_. Association for Computing Machinery, New York, NY, USA, 70–74.  https://doi.org/10.1145/3397482.3450721
  * Piao and Breslin (2016) Guangyuan Piao and John G. Breslin. 2016.  Exploring Dynamics and Semantics of User Interests for User Modeling on Twitter for Link Recommendations. In _Proceedings of the 12th International Conference on Semantic Systems_ (Leipzig, Germany) _(SEMANTiCS 2016)_. Association for Computing Machinery, New York, NY, USA, 81–88.  https://doi.org/10.1145/2993318.2993332
  * Piech et al. (2015) Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015.  Deep Knowledge Tracing. In _Advances in Neural Information Processing Systems_.
  * Ponza et al. (2020) Marco Ponza, Paolo Ferragina, and Soumen Chakrabarti. 2020\.  On Computing Entity Relatedness in Wikipedia, with Applications.  _Knowledge-Based Systems_ 188 (2020).  https://doi.org/10.1016/j.knosys.2019.105051
  * Ramesh et al. (2014) Arti Ramesh, Dan Goldwasser, Bert Huang, Hal Daume III, and Lise Getoor. 2014.  Learning latent engagement patterns of students in online courses. In _Proc. of AAAI Conference on Artificial Intelligence_.
  * Rasch (1960) G.E. Rasch. 1960\.  _Probabilistic Models for Some Intelligence and Attainment Tests_. Vol. 1.
  * Schenkel et al. (2007) Ralf Schenkel, Andreas Broschart, Seungwon Hwang, Martin Theobald, and Gerhard Weikum. 2007\.  Efficient text proximity search. In _International Symposium on String Processing and Information Retrieval_. Springer, 287–299.
  * Selent et al. (2016) Douglas Selent, Thanaporn Patikorn, and Neil Heffernan. 2016\.  ASSISTments Dataset from Multiple Randomized Controlled Experiments. In _Proc. of the Third (2016) ACM Conf. on Learning @ Scale_ _(L@S ’16)_.  https://doi.org/10.1145/2876034.2893409
  * Tatsuoka (1983) Kikumi K Tatsuoka. 1983\.  Rule space: An approach for dealing with misconceptions based on item response theory.  _Journal of educational measurement_ (1983), 345–354.
  * Verma et al. (2021) Gaurav Verma, Trikay Nalamada, Keerti Harpavat, Pranav Goel, Aman Mishra, and Balaji Vasan Srinivasan. 2021.  Non-Linear Consumption of Videos Using a Sequence of Personalized Multimodal Fragments. In _26th International Conference on Intelligent User Interfaces_ (College Station, TX, USA) _(IUI ’21)_. Association for Computing Machinery, New York, NY, USA, 249–259.  https://doi.org/10.1145/3397481.3450672
  * Vie and Kashima (2019) Jill-Jênn Vie and Hisashi Kashima. 2019.  Knowledge tracing machines: Factorization machines for knowledge tracing. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 750–757.
  * Walkington et al. (2019) Candace Walkington, Virginia Clinton, and Anthony Sparks. 2019\.  The effect of language modification of mathematics story problems on problem-solving in online homework.  _Instructional Science_ 47, 5 (2019), 499–529.
  * Wang et al. (2020) Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, Jose Miguel Hernandez Lobato, Richard Turner, Richard Baraniuk, Craig Barton, Simon Peyton Jones, Simon Woodhead, and Cheng Zhang. 2020.  Diagnostic Questions: The NeurIPS 2020 Education Challenge.  https://www.microsoft.com/en-us/research/publication/diagnostic-questions-the-neurips-2020-education-challenge/
  * Wang et al. (2021) Zichao Wang, Sebastian Tschiatschek, Simon Woodhead, José Miguel Hernández-Lobato, Jones Simon Peyton, Richard G. Baraniuk, and Cheng Zhang. 2021.  Educational Question Mining At Scale: Prediction, Analysis and Personalization. In _Symposium on Educational Advances in Artificial Intelligence (AAAI-EAAI)_.  https://www.microsoft.com/en-us/research/publication/educational-question-mining-at-scale-prediction-analysis-and-personalization/
  * Wikimedia (2018) Wikimedia. 2018\.  _Wikimedia Downloads_.
  * Wilson et al. (2016) Kevin H Wilson, Yan Karklin, Bojian Han, and Chaitanya Ekanadham. 2016.  Back to the basics: Bayesian extensions of IRT outperform neural networks for proficiency estimation.  (2016).
  * Wu et al. (2018) Siqi Wu, Marian-Andrei Rizoiu, and Lexing Xie. 2018\.  Beyond Views: Measuring and Predicting Engagement in Online Videos. In _Proc. of the Twelfth Int. Conf. on Web and Social Media_.
  * Yano and Kang (2016) Tae Yano and Moonyoung Kang. 2016.  _Taking advantage of wikipedia in natural language processing_.  Technical Report. Technical report, Carnegie Mellon University Language Technologies Institute.
  * Yu et al. (2020) Jifan Yu, Gan Luo, Tong Xiao, Qingyang Zhong, Yuquan Wang, Wenzheng Feng, Junyi Luo, Chenyu Wang, Lei Hou, Juanzi Li, et al. 2020\.  MOOCCube: a large-scale data repository for NLP applications in MOOCs. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 3135–3142.
  * Yudelson et al. (2013) Michael V. Yudelson, Kenneth R. Koedinger, and Geoffrey J. Gordon. 2013.  Individualized Bayesian Knowledge Tracing Models. In _Proc. of Artificial Intelligence in Education_ , H. Chad Lane, Kalina Yacef, Jack Mostow, and Philip Pavlik (Eds.).