# Understanding tree: a tool for estimating an individual’s understanding of
conceptual knowledge

Gangli Liu

Tsinghua University  Beijing 100084, China<EMAIL_ADDRESS>
###### Abstract

People learn whenever and wherever possible, and whatever they like or
encounter–Mathematics, Drama, Art, Languages, Physics, Philosophy, and so on.
With the bursting of knowledge, evaluation of one’s understanding of
conceptual knowledge becomes increasingly difficult. There are a lot of
demands for evaluating one’s understanding of a piece of knowledge, e.g.,
facilitating personalized recommendations; discovering one’s expertises and
deficiencies in a field; recommending topics for a conversation between people
with different educational or cultural backgrounds in their first encounter;
recommending a learning material to practice a meaningful learning etc.
Assessment of understanding of knowledge is conventionally practiced through
tests or interviews, but they have some limitations such as low-efficiency and
not-comprehensive. We propose a method to estimate one’s understanding of
conceptual knowledge, by keeping track of his/her learning activities. It
overcomes some limitations of traditional methods, hence complements
traditional methods.

##  1 Introduction

Our world is bursting with knowledge. Nearly every discipline has been
subdivided into numerous sub-disciplines. By July 27 2017, there are 42.5
million entries on
Wikipedia111https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia, which,
undoubtedly, stands for only a small portion of human knowledge. People learn
whenever and wherever possible. People’s learning of knowledge is not confined
to childhood or the classroom but takes place throughout life and in a range
of situations; it can take the form of formal learning or informal learning
[42], such as daily interactions with others and with the world around us.
Lifelong learning is the “ongoing, voluntary, and self-motivated” pursuit of
knowledge for either personal or professional reasons [11]. According to
Tough’s study, almost 70% of learning projects are self-planned [53].

As people learn eternally, one significant issue is to evaluate how much
knowledge an individual is possessing at a particular time. E.g., suppose we
have a large database that records all the entries of Wikipedia and a person’s
understanding degree of each entry (on a scale of 1 to 10). With this
information, a lot of new applications are becoming practical. The following
are some examples:

#### 1\. Determine a person’s knowledge state

If we set a threshold to the understanding grades, and assume the subject has
understood a knowledge entry if its grade is larger than the threshold, then
we can determine a person’s knowledge state and knowledge composition at a
particular time. As Figure 1 illustrates.

Figure 1: A person’s knowledge composition

#### 2\. Discover a person’s expertises and deficiencies

Expertise finding is critical for an organization or project. Since the
participation of experts plays an important role for the success of an
organization or project. With the understanding evaluation database, it is
convenient to discover a person’s domain-level expertise and topic-level
expertise. E.g., if a person has a question of what is Poincare Conjecture, he
do not need to ask a mathematician (domain-level expert), who is not always
available. Instead, he can ask one of his friends who is not a mathematician
but has topic-level expertise in it. Besides expertise, we can also discover a
person’s deficiencies in a field, so he can remedy the deficiencies.

#### 3\. Make personalized recommendations

If we know a person’s understanding degree to each piece of knowledge, it is
natural to make personalized recommendations to the subject based on the
information. E.g., suppose we know a person has good understanding in the
topics of Deep Learning, then we can recommend latest papers about Deep
Learning to the subject, or recommend the person as a reviewer of a paper
related to Deep Learning. In addition, we can recommend learning materials to
the subject to help him practice meaningful learning, which will be discussed
in details in Section 4.

###  1.1 Procedural and Conceptual Knowledge

Studies of knowledge indicate that knowledge may be classified into two major
categories: procedural and conceptual knowledge [20, 37]. Procedural knowledge
is the knowledge exercised in the accomplishment of a task, and thus includes
knowledge which cannot be easily articulated by the individual, since it is
typically nonconscious (or tacit). It is commonly referred to as “know-how”.
Such as knowing how to cook delicious food, how to drive an airplane, or how
to play basketball etc. Conceptual knowledge is quite different from
procedural knowledge. It involves understanding of the principles that govern
a domain and of the interrelations between pieces of knowledge in a domain; it
concerns understanding and interpreting concepts and the relations between
concepts. It is commonly referred to as “know-why”, such as knowing why
something happens in a particular way. In this article, we only deal with
evaluating a person’s understanding of conceptual knowledge.

###  1.2 The framework

At present, assessment of one’s understanding of conceptual knowledge is
primarily through tests [43, 22, 45] or interviews, which have some
limitations such as low-efficiency and not-comprehensive. E.g., it needs other
people’s cooperation to accomplish the assessment, which is time-consuming;
moreover, only a small portion of topics of a domain is evaluated during an
assessment, which cannot comprehensively reflect a person’s knowledge state of
the domain.

We propose a new method named Individual Conceptual Knowledge Evaluation Model
(ICKEM) to evaluate one’s understanding of a piece of conceptual knowledge
quantitatively. It has the advantage of evaluating a person’s understanding of
conceptual knowledge independently, comprehensively, and automatically. It
keeps track of one’s daily learning activities (reading, listening,
discussing, writing etc.), dividing them into a sequence of learning sessions,
then analyzes the text content of each learning session to discover the
involved knowledge topics and their shares in the session. Then a learning
experience is inserted to the involved knowledge topics’ learning histories
(It maintains a leaning history for _each_ knowledge topic). Therefore, after
a period of time (e.g., several years or decades of years), a knowledge
topic’s leaning history that records the subject’s each leaning experience
about the topic is generated. Based on the learning history, the subject’s
familiarity degree to a knowledge topic is evaluated. Finally, it estimates
one’s understanding degree to a topic, by comprehensively evaluating one’s
familiarity degrees to the topic itself and other topics that are essential to
understand the topic. Figure 2 is the framework of ICKEM. Each hexagon of the
diagram indicates a processing step; the following rectangle indicates the
results of the process.

Figure 2: The framework of ICKEM

The remainder of this paper is organized as follows. Section 2 discusses how
to calculate a person’s familiarity degree to a knowledge topic. Section 3
introduces a method to estimate a person’s understanding degree to a topic, by
checking the familiarity degrees of the topic’s understanding tree. It also
presents an algorithm to generate a knowledge topic’s understanding tree. In
Section 4, we introduce an utilization of ICKEM, computer-aided incremental
meaningful learning (CAIML), which helps an individual practice meaningful
learning. Section 5 discusses related issues about evaluating one’s
understanding of conceptual knowledge. We cover related work in Section 6,
before concluding in Section 7.

##  2 Evaluate familiarity degree

This section introduces the procedures that are devised to evaluate a person’s
familiarity degree to a knowledge topic. It starts by presenting a formal
definition of knowledge and learning, then discusses how to divide a person’s
daily learning activities into a series of learning sessions, and analyze the
text learning content to obtain a topic’s share in a session. After these
procedures, a knowledge topic’s learning history can be generated. Finally,
based on the learning history, the subject’s familiarity degree to a topic is
calculated.

###  2.1 Definition of knowledge and learning

Knowledge is conventionally defined as beliefs that are true and justified. To
be ‘true’ means that it is in accord with the way in which objects, people,
processes and events exist and behave in the real world. However, exactly what
evidence is necessary and sufficient to allow a true belief to be ‘justified’
has been a topic of discussion (largely among philosophers) for more than
2,000 years [22]. In ICKEM, a Knowledge Point is defined as a piece of
conceptual knowledge that is explicitly defined and has been widely accepted,
such as Bayes’ theorem, Euler’s formula, mass-energy equivalence, Maxwell’s
equations, gravitational wave, and the expectation-maximization algorithm etc.

Learning is the process of acquiring, modifying, or reinforcing knowledge,
behaviors, skills, values, or preferences in memory [52, 22, 7]. An
individual’s possessing of knowledge is the product of all the experiences
from the beginning of his/her life to the moment at hand [22, 4]. Learning
produces changes in the organism and the changes produced are relatively
permanent [47].

###  2.2 Discriminate learning sessions

In ICKEM, a person’s daily activities are classified into two categories:
learning activities and non-learning activities. An activity is recognized as
a learning activity if its content involves at least one Knowledge Point of a
predefined set. In addition, the learning activities are divided into a
sequence of learning sessions, since it is essential to know how many times
and how long for each time the individual has learned a Knowledge Point. An
individual can employ many methods to learn conceptual knowledge, such as
reading, listening, discussing, and writing. Different strategies should be
used to discriminate learning sessions for different learning methods. E.g.,
for leaning by reading documents on a computer, Algorithm 1 is devised to
discriminate learning sessions. For other learning methods, it is more
complicated to divide learning sessions. However, there are already some
attempts for detecting human daily activities [40, 44, 50]. Sung et al.
devised an algorithm for recognizing human daily activities from RGB-D Images
[50]. They tested their algorithm on detecting and recognizing twelve
different activities (brushing teeth, cooking, working on computer, talking on
phone, drinking water, talking on a chair etc.) performed by four people and
achieved good performance.

Algorithm 1 works by periodically checking the occurrence of the following
events:

  1. 1.

A document is opened;

  2. 2.

The foreground window has switched to an opened document from another
application (APP), such as a game APP;

  3. 3.

After the computer being idled for a period of time, there are some mouse or
keyboard inputs detected, which indicates the individual has come back from
other things. Meanwhile, the foreground window is an opened document;

  4. 4.

A document is closed;

  5. 5.

The foreground window has switched to another APP from a document;

  6. 6.

The foreground window is a document, but the computer has idled for a certain
period of time without any mouse or keyboard inputs detected, the individual
is assumed to have left to do other things.

Occurrence of Event 1, 2, or 3 indicates a learning session has started; if
Event 4, 5, or 6 occurred, a learning session is assumed being terminated. The
duration of a learning session equals the interval between its start and stop
time.

Algorithm 1 An algorithm to discriminate one’s learning sessions when reading.

1: while The Reader APP is running do

2: if Event 4 OR Event 5 OR Event 6 occurred then

3: Record that a document’s learning session has stopped;

4: else

5: if Event 1 OR Event 2 OR Event 3 occurred then

6: Record that a learning session about the current document has started;

7: end if

8: else

9: if There is no APP and document switch then

10: Check and record if there is a Page switch;

11: end if

12: end if

13: Keep silent for $T$ seconds;

14: end while

Figure 3 shows some examples of discriminated learning sessions. Attribute
“did” means document ID, which indexes a document uniquely. Attribute
“actiontype” indicates the type of an action. “Doc Act” means a document has
been activated. “Page Act” is defined similarly. “Doc DeAct” means a document
has been deactivated. That is to say, a learning session has stopped.
Attribute “page” indicates a page number. Attribute “duration” records how
long a page has been activated in seconds. If two learning sessions’ interval
is less than a certain threshold, such as 30 minutes, and their learning
material is the same (e.g., the same document), they are merged into one
session. Therefore, “Session 2” and “Session 3” are merged into one session.

Figure 3: Some examples of discriminated learning sessions.

###  2.3 Capture the text learning content

Most learning processes are associated with a piece of learning material.
E.g., reading a book, the book is the learning material; attending a course or
discussion, the course and discussion contents can be regarded as the learning
material. Some learning materials are text or can be converted to text. E.g.,
discussing a piece of knowledge with others. The discussion contents can be
converted to text by exploiting Speech Recognition. Similarly, if one is
reading a printed book, the contents of the book can be captured by wearable
computers like Google Glass and then converted to text through Optical
Character Recognition (OCR). If the book is electronic, no conversion is
needed; the text can be extracted directly. Since Algorithm 1 records the
accurate set of pages a person has read during a learning session, only the
text content of related pages is extracted.

###  2.4 Calculate a Knowledge Point’s share

A learning session may involve many topics, it is necessary to know how much a
learning session involves a topic. To calculate a Knowledge Point’s share,
maybe the simplest way is to deem the text learning content as a bag of words,
and calculate a Knowledge Point’s share based on its Term Frequency (TF) or
normalized TF. Just like Equation 1 and 2. $N_{i}$ is term $i$’s normalized
TF. It is calculated with Equation 1, where $T_{i}$ is term $i$’s TF,
$Max(TF)$ is the maximum TF of the captured text content, $\alpha$ is a
constant regulatory factor.

$N_{i}=\alpha+(1-\alpha)*T_{i}/Max(TF)$ (1)
$\xi_{i}=\frac{N_{i}}{\sum_{j=1}^{m}N_{j}}$ (2)

Another method of discovering a Knowledge Point’s share is to analyze the text
learning content with topic model. A topic model is a type of statistical
model for discovering the abstract ”topics” that occur in a collection of
documents. Topic model is a frequently used text-mining tool for discovery of
hidden semantic structures in a text body [5, 49]. An early topic model was
described in [41]. Another one, called probabilistic latent semantic analysis
(PLSA), was created by Hofmann in 1999 [21]. Latent Dirichlet allocation (LDA)
is the most common topic model currently in use [6]. It is a generalization of
PLSA. It introduces sparse Dirichlet prior distributions over document-topic
and topic-word distributions. Other topic models are generally extensions on
LDA.

The inputs of a probabilistic topic model are a collection of $N$ documents, a
vocabulary set $V$, and the number of topics $k$. The outputs of a
probabilistic topic model are the following:

  * •

k topics, each is word distribution : $\\{\theta_{1},...,\theta_{k}\\}$;

  * •

Coverage of topics in each document $d_{i}$: $\\{\pi_{i1},...,\pi_{ik}\\}$;
$\pi_{ij}$ is the probability of document $d_{i}$ covering topic $\theta_{j}$.

The subject’s $N$ learning sessions’ text-contents can be deemed as a
collection of $N$ documents, then the document collection is analyzed with a
topic model like LDA. Based on the outputs of topic model, Equation 3 is used
to calculate a Knowledge Point’s share in a learning session. Only the top $m$
terms of each topic are considered. For document $d$, $\varphi_{ij}$ is the
share of term $i$ which comes from topic $j$, $\pi_{j}$ is topic $j$’s share
of document $d$, $p(t_{i}|\theta_{j})$ is term $i$’s share of topic $j$. A
Knowledge Point’s share equals its term share. Therefore, with topic model,
Knowledge Points involved in each learning session can be discovered; their
shares can also be calculated.

$\varphi_{ij}=\frac{\pi_{j}p(t_{i}|\theta_{j})}{\sum_{j=1}^{k}\sum_{i=1}^{m}\pi_{j}p(t_{i}|\theta_{j})}$
(3)

###  2.5 The subject’s state during a session

The subject’s physical and psychological status may influence the effect of a
learning session. E.g., if the subject is tired, or severely injured, the
learning effect is usually worse than being in a normal state. Similarly, if
the subject is in a state of severe depression, anxiety, or scatterbrained,
the learning effect cannot be good either. One issue is how to collect values
for these attributes. Fortunately, there are already systems for monitoring a
person’s physical and psychological status [56, 9, 13, 25, 29]. Another issue
is studies need to be conducted to decide how these attributes would affect
the learning effect. Preferable to have some equations to calculate a
“physical and psychological status factor”, describing how the learning effect
is affected by the subject’s physical and psychological state during a
session.

On the other hand, if we just want a rough estimation of the subject’s
familiarity degree to a Knowledge Point, these physical and psychological
status attributes may be ignored. Since we are observing the subject in a long
time rang, typically several years or decades of years, we can assume the
subject is in “normal” state most of the time, and ignore its fluctuation.

###  2.6 Effectiveness of a learning method

Different learning methods may have different levels of effectiveness. E.g.,
learning by reading a book and learning by discussing with others, do these
two methods have equivalent levels of memory retention after the learning
experience? (supposing other conditions are equivalent, such as learning
content, session duration, physical and psychological status etc.) These is no
consensus about the effectiveness of different learning methods. The National
Training Laboratories (NTL) Institute proposes a _learning pyramid_ model,
which maps a range of learning methods onto a triangular image in proportion
to their effectiveness in promoting student retention of the material taught
[48, 32]. However, the credibility and research base of the model have been
questioned by other researchers [26, 27]. Further research is necessary to
make a systematic comparison between the effectiveness of different learning
methods. Preferable to have a “learning method factor” depicting the
effectiveness of a method. Similarly, if we just want a rough estimation of
the subject’s familiarity degrees, we can omit the difference among different
learning methods. As [26] concluded, there is no learning method being
consistently superior to the others and all being effective in certain
contexts. [26] also stressed the importance of reading: reading is not only an
effective learning method, it is also the main foundation for becoming a
“life-long learner”.

###  2.7 A Knowledge Point’s learning history

With the discriminated learning sessions, a Knowledge Point’s share in each
session, the subject’s physical and psychological status during a session, and
the “learning method factor”, after a period of time, the subject’s learning
histories about each Knowledge Point can be generated. Figure 4 shows an
exemplary learning history. It records a person’s each learning experience
about a Knowledge Point. “LCT” stands for “learning cessation time”. It is
used to calculate the interval between the learning time and the evaluation
time, which is then used to estimate how much information has been lost due to
memory decay. “Duration” is the length of a learning session. “Proportion” is
the Knowledge Point’s share during a learning session. “PPS factor” stands for
the “physical and psychological status factor”. It is a number between 0 and 1
that is calculated based on the subject’s average physical and psychological
status during a session. “LM factor” stands for the “learning method factor”.
It is also a number between 0 and 1 that is allocated to a learning method
according to its effectiveness level.

Figure 4: An exemplary learning history of a Knowledge Point

###  2.8 Memory retention of a learning experience

People learn all the time; meanwhile, people forget all the time. Human memory
declines over time. Interestingly, most researchers report there is no age
difference for memory decay [46]. To calculate the familiarity degree, we need
to address how the effect of a learning experience decays over time. However,
there is no consensus of how human memory decays. Psychologists have suggested
many functions to describe the monotonic loss of information with time [46].
But there is no unanimous agreement of which one is the best. The search for a
general description of forgetting is one of the oldest unresolved problems in
experimental psychology [3].

We propose to use Ebbinghaus’ forgetting curve equation [14] to describe the
memory retention of a learning experience over time. Since it is the most
well-known forgetting curve equation and its soundness has been proved by many
studies [39, 15, 19]. Ebbinghaus found Equation 4 can be used to describe the
proportion of memory retention after a period of time222It can be found at
http://psychclassics.yorku.ca/Ebbinghaus/memory7.htm, where $t$ is the time in
minutes counting from one minute before the end of the learning, $k$ and $c$
are two constants that equal 1.84 and 1.25, respectively. Figure 5 shows the
percentage of memory retention over time calculated by Equation 4. The Y axis
is the percentage of memory retention; the X axis is the time-since-learning
in minutes. It can be seen that memory retention declines drastically during
the first 24 hours (1,440 minutes), then the speed tends to be steady.

$b(t)=k/((\log t)^{c}+k)$ (4)

Figure 5: The percentage of memory retention over time calculated by Equation
4

###  2.9 Calculate the familiarity degree

Based on the learning history, Equation 5 is utilized to calculate the
subject’s Familiarity Measure to a Knowledge Point $k_{i}$ at time $t$. A
Familiarity Measure is defined as a score that depicts a person’s familiarity
degree to a Knowledge Point. Its unit is defined as _gl_. The input is
$k_{i}$’s learning history–a sequence of $m$ learning sessions (like Figure
4). $d_{j}$ is session $j$’s duration; $\xi_{ij}$ is Knowledge Point $k_{i}$’s
share in session $j$; $t_{j}$ is session $j$’s “learning cessation time”,
$b(t-t_{j})$ calculates the percent of memory retention of session $j$ at time
$t$ with Equation 4; $F^{pps}_{j}$ is the “physical and psychological status
factor” of session $j$; $F^{lm}_{j}$ is the “learning method factor” of
session $j$.

$F(k_{i},t)=\sum_{j=1}^{m}d_{j}*\xi_{ij}*b(t-t_{j})*F^{pps}_{j}*F^{lm}_{j}$
(5)

The computation hypothesizes each learning experience about a Knowledge Point
contributes some effect to the subject’s current understanding of it, and the
learning effect declines over time according to Ebbinghaus’ forgetting curve.
Other attributes (the subject’s physical and psychological status, learning
method) that may affect the learning effect are counted in as numeric factors.
The Familiarity Measure of a Knowledge Point is calculated as the additive
effects of all the learning experiences about it. If we omit the influence of
subject physical and psychological status and the difference among learning
methods, Equation 5 can be simplified to Equation 6.

$F(k_{i},t)=\sum_{j=1}^{m}d_{j}*\xi_{ij}*b(t-t_{j})$ (6)

Table 1 compares Familiarity Measures calculated by Equation 5 and Equation 6
at different times based on the learning history of Figure 4. The evaluation
times are an hour, a day, a month, a year, and 10 years after last learning of
the knowledge point.

| 5/17/2016 16:25 | 5/18/2016 15:25 | 6/17/2016 15:25 | 5/17/2017 15:25 | 5/17/2026 15:25
---|---|---|---|---|---
Equation 5 | 25 | 23.5 | 21.9 | 19.4 | 16.6
Equation 6 | 42.6 | 40.2 | 37.6 | 33.3 | 28.5
Table 1: Familiarity Measures calculated at different times

##  3 Estimate understanding degree

Understanding is quite subtle to measure. We hypothesize that if a person is
familiar with a Knowledge Point itself and all the background knowledge that
is essential to understand it, he should have understood the Knowledge Point.
Because Familiarity Measure depicts the cumulated effects of one’s learning
experiences about a topic, high levels of Familiarity Measures imply intensive
learning activities about the suite of knowledge topics. Intensive learning
activities usually result in a good understanding.

The background Knowledge Points that are essential to understand a Knowledge
Point can be extracted by analyzing its definition. Table 2 lists eight
reduced documents, each of them is a definition of a Knowledge Point in
Probability Theory or Stochastic Process, the texts are quoted from Wikipedia
and other websites. The third column of Table 2 lists the involved Knowledge
Points in the documents, which are deemed as the background knowledge to
understand the host Knowledge Point.

Doc |  Content |  Knowledge Points
---|---|---
D1 |  A Strictly Stationary Process (SSP) is a Stochastic Process (SP) whose Joint Probability Distribution (JPD) does not change when shifted in time. |  SSP, JPD, Time, SP
D2 |  A Stochastic Process (SP) is a Probability Model (PM) used to describe phenomena that evolve over time or space. In probability theory, a stochastic process is a Time Sequence (TS) representing the evolution of some system represented by a variable whose change is subject to a Random Variation (RaV). |  SP, PM, TS, Time, Space, System, Variable, RaV
D3 |  In the study of probability, given at least two Random Variables (RV) X, Y, … that are defined on a Probability Space (PS), the Joint Probability Distribution (JPD) for X, Y, … is a Probability Distribution (PD) that gives the probability that each of X, Y, … falls in any particular range or discrete set of values specified for that variable. |  JPD, RV, PS, PD, Variable, Probability
D4 |  A Probability model (PM) is a mathematical representation of a random phenomenon. It is defined by its Sample Space (SS), events within the SS, and probabilities associated with each event. |  PM, SS, Event, Probability
D5 |  In probability and statistics, a Random variable (RV) is a variable quantity whose possible values depend, in some clearly-defined way, on a set of random events. |  RV, Variable, Event
D6 |  A Probability Space (PS) is a Mathematical Construct (MC) that models a real-world process consisting of states that occur randomly. It consists of three parts: a Sample Space (SS), a set of events, and the assignment of probabilities to the events. |  PS, MC, SS, Probability, Event
D7 |  A Probability Distribution (PD) is a table or an equation that links each outcome of a statistical experiment with its probability of occurrence. |  PD, Probability
D8 |  The Sample Space (SS) is the set of all possible outcomes of the samples. |  SS, Sample
Table 2: A list of documents and their involved Knowledge Points

An Understanding Tree is a treelike data structure which compiles the
background Knowledge Points that are essential to understand the root
Knowledge Point. Figure 6 illustrates four Understanding Trees based on the
definitions of Table 2. The nodes of the tree can be further interpreted by
other Knowledge Points until they are Basic Knowledge Points (BKP). A BKP is a
Knowledge Point that is simple enough so that it is not interpreted by other
Knowledge Points. Figure 7 shows a fully extended Understanding Tree based on
the definitions of Table 2. Figure 8 shows an exemplary Understanding Tree
with all the redundant nodes eliminated. Each node is tagged with a
Familiarity Measure calculated with Equation 5 or 6. The leaf nodes of Figure
7 and Figure 8 are BKPs. The height and number of nodes of an Understanding
Tree characterize the complexity degree of it. Understanding Tree can be used
to evaluate a person’s topic-level expertise, such as evaluating a person’s
understanding to a specific algorithm like the Quicksort algorithm.

Figure 6: Some examples of Understanding Trees Figure 7: A fully extended
Understanding Tree Figure 8: A standard Understanding Tree tagged with
Familiarity Measures Figure 9: Familiarity Measures transformed into
percentages

###  3.1 Calculation of understanding degree

If all the Familiarity Measures of an Understanding Tree exceed a threshold
(such as 100), it is assumed that the person has understood the root Knowledge
Point. Then the Knowledge Point is classified as “Understood”; otherwise, it
is classified as “Not Understood”. Due to the differences of people’s
intelligence and talent, different people may have different thresholds.

If a Familiarity Measure is less than the threshold, a percentage is
calculated by dividing it by the threshold, indicating the subject’s percent
of familiarity of the node; if the Familiarity Measure is greater than the
threshold, the percentage is set to 1, implying the quantity of familiarity of
this node is large enough for understanding the root, extra familiarity is
good but not necessary. Equation 7 is used to calculate the percentage of
familiarity, $F(k_{i},t)$ is the subject’s Familiarity Measure to Knowledge
Point $k_{i}$ at time $t$, $f_{T}$ is the threshold.

If a Knowledge Point is classified as “Not Understood”, a percent of
understanding is calculated with Equation 8. $PU(k_{r},t)$ is the subject’s
percent of understanding of the root Knowledge Point at time $t$,
$PF(k_{r},t)$ is the percent of familiarity of the root,
$\frac{1}{m}\sum_{j=1}^{m}PF(k_{j},t)$ calculates the average percent of
familiarities of its descendants (not including the root). E.g., Figure 9 is
the Understanding Tree of Figure 8 with Familiarity Measures transformed into
percentages (assuming $f_{T}$ equals 100). The $PF(k_{r},t)$ of it equals 85%,
and $\frac{1}{m}\sum_{j=1}^{m}PF(k_{j},t)$ equals 89%, so the $PU(k_{r},t)$
equals 76%, indicating the subject’s understanding of the root Knowledge Point
is 76%. If $PU(k_{r},t)$ is less than 100%, the Knowledge Point is classified
as “Not Understood”. Thus it can be seen that we are using a conservative
strategy for estimating the subject’s understanding. For a Knowledge Point to
be classified as “Understood”, the subject must be familiar with _every_ node
of its Understanding Tree. This strategy should guarantee a good precision of
the Knowledge Points that are classified as “Understood”, and thus guarantees
the quality of the estimated topic-level expertise of the subject.

$PF(k_{i},t)=\left\\{\begin{array}[]{cl}1&F(k_{i},t)\geq f_{T}\\\
F(k_{i},t)/f_{T}&F(k_{i},t)<f_{T}\end{array}\right.$ (7)

$PU(k_{r},t)=PF(k_{r},t)*\frac{1}{m}\sum_{j=1}^{m}PF(k_{j},t)$ (8)

If $PU(k_{r},t)$ equals 100%, the subject is assumed having understood the
root Knowledge Point, then the average Familiarity Measure of the
Understanding Tree divided by the threshold features the magnitude of
understanding. Since people usually are very familiar with the BKPs, it may be
preferable to exclude them or normalize their effects when computing the
average Familiarity Measure.

###  3.2 Construction of Understanding Tree

An Understanding Tree can be constructed manually by a group of experts, or
generated automatically by machines. Algorithm 2 shows the steps to generate
an Understanding Tree automatically. Table 3 illustrates three definitions of
the Central Limit Theorem (CLT)333They can be found at
http://www.math.uah.edu/stat/sample/CLT.html,
http://sphweb.bumc.bu.edu/otlt/MPH-
Modules/BS/BS704_Probability/BS704_Probability12.html,
http://stattrek.com/statistics/dictionary.aspx, the third column lists the
involved Knowledge Points in each definition. According to the rule mentioned
in Algorithm 2, the child nodes of CLT are selected as _sample_ ,
_distribution_ , _mean_ , _independent_ , and _normal_ (Knowledge Points that
are not BKPs can be further extended). To be safe, the generated Understanding
Tree can be inspected by human experts. Understanding Tree is a static
structure, once constructed, it is not likely to change them.

Algorithm 2 An algorithm to construct Understanding Tree

0: The set of all Knowledge Points, $\Omega$;The set of all BKPs, $B$;The root
Knowledge Point, $R_{k}$;

0: $R_{k}$’s Understanding Tree;

1: Search definitions of $R_{k}$ from a library of authoritative documents;

2: Discover involved Knowledge Points for each definition;

3: Select Knowledge Points according to some rules, e.g., more than half of
the definitions have referenced the Knowledge Point;

4: Recursively extend non-BKP Knowledge Points that are selected;

5: return All the selected Knowledge Points;

|  Content |  Knowledge Points
---|---|---
1 |  The Central Limit Theorem (CLT) states that the sampling distribution of the mean of any independent, random variable will be normal or nearly normal, if the sample size is large enough. |  CLT, sample, distribution, mean, independent, random variable, normal, size
2 |  The Central Limit Theorem (CLT) states that the distribution of the sum (or average) of a large number of independent, identically distributed variables will be approximately normal, regardless of the underlying distribution. |  CLT, distribution, sum, average, independent, variable, normal
3 |  The Central Limit Theorem (CLT) states that if you have a population with mean $\mu$ and standard deviation $\sigma$ and take sufficiently large random samples from the population with replacement, then the distribution of the sample means will be approximately normally distributed. |  CLT, population, standard deviation, random, replacement, distribution, sample, mean, normal
Table 3: Three definitions of the Central Limit Theorem (CLT)

##  4 Facilitate meaningful learning

Knowing one’s understanding degrees of knowledge helps practice meaningful
learning. Meaningful learning is the concept that learned knowledge is fully
understood to the extent that it relates to other knowledge, implies there is
a comprehensive knowledge of the context of the facts learned [36]. ICKEM
helps people practice Computer-aided Incremental Meaningful Learning (CAIML).
CAIML is defined as a strategy of letting the computer estimate a person’s
current knowledge states, then recommend the optimum learning material for the
subject to accomplish a meaningful learning. The optimum material introduces
some new knowledge blended with old knowledge the subject has known,
meanwhile, interpreting the new knowledge.

For example, a college student, who has comprehended the basic knowledge of
Advanced Mathematics and Computer Science, wants to be an expert of Artificial
Intelligence (AI) by teaching himself. A professor recommends him 1,000
documents related to AI, and asserts if he can fully understand the contents
of the documents, he will be an expert of AI. The question is: what is the
best sequence for the student to learn the documents? Some documents are easy
to understand, and should be read in the beginning; some documents are
intricate, learning them in the first place is painful and frustrating, and
should be put at the end of the learning process. If a computer knows a
person’s knowledge states at any time, it is not difficult to make the
learning plan, and recommend the optimum document for current learning.

###  4.1 An algorithm to facilitate CAIML

Algorithm 3 is devised to facilitate CAIML. It recommends the optimum document
for the subject to learn, by analyzing his current knowledge states. It
searches for the document which has the least Knowledge Points that are not
understood by the subject, which implies it is the easiest document to
understand at present. In the document, the “Understood” Knowledge Points
server as interpretations to the “Not Understood” ones when learning the
document. Therefore, the algorithm facilitates a meaningful learning.

Algorithm 3 An algorithm to recommend the optimum document(s) for CAIML

0: The set of documents to be learned;The set of all Knowledge Points, tagged
with the subject’s Familiarity Measures to them;The set of all non-BKP
Knowledge Points’ Understanding Trees;

0: The optimum document(s) for current learning to practice a meaningful
learning;

1: Extract involved Knowledge Points in each document, classify them as
“Understood” and “Not Understood”, according to the rule mentioned in Section
3.1;

2: Count the number of “Not Understood” Knowledge Points for each document;

3: return The document(s) with the least “Not Understood” Knowledge Points;

An alternative method to recommend the optimum document is to estimate the
subject’s anticipated percentage of understanding of each document, then
return the one that is most approaching 100%. A person’s understanding of a
document is calculated with Equation 9. Suppose the document involves $n$
Knowledge Points, $PU(d,t)$ is the percent of understanding of the document at
time $t$; $\xi_{i}$ is Knowledge Point $k_{i}$’s share of the document, its
calculation has been discussed in Section 2.4; $PU(k_{i},t)$ is Knowledge
Point $k_{i}$’s percent of understanding at time $t$. If a person has
understood all the Knowledge Points the document contains, its $PU(d,t)$ is
100%. It is possible that a person learns a “100% understood” document to
strengthen his knowledge. As Equation 9 can estimate a person’s understanding
degree to document, it can also be used to recommend appropriate reviewers for
a paper.

$PU(d,t)=\frac{\sum_{i=1}^{n}\xi_{i}*PU(k_{i},t)}{\sum_{i=1}^{n}\xi_{i}}$ (9)

People’s memory fades away over time, the Familiarity Measures decrease
accordingly. It is also possible that the computer recommends a document that
had been tagged “100% understood”, because the $PU(d,t)$ has abated.

###  4.2 An example of CAIML

Here is an example to illustrate the logic of CAIML. Suppose a person wants to
fully understand the suite of documents listed in Table 2 by learning them.
Figure 7 shows all the Knowledge Points involved in the documents, the third
column of Table 2 lists ones referenced by each of them. The subject is
assumed to have understood the BKPs before the beginning of the learning,
which are the leaf nodes of Figure 7 (if not, he can learn them first). Table
4 shows the number of “Not Understood” Knowledge Points before the beginning
of the learning and after each learning, for each document. E.g., there are 8
“Not Understood” Knowledge Points before the starting of the learning for D1,
they are SSP, SP, JPD, PM, SS, RV, PS, and PD. According to Algorithm 3, the
subject should first learn either of D5, D7 or D8. After learning of D5, the
subject is assumed to have understood RV. Therefore, the number of “Not
Understood” Knowledge Points for D1 becomes 7. The first column of Table 4
suggests one of the optimum learning sequences of the documents, which is D5,
D8, D4, D2, D7, D6, D3, and D1.

| D1 | D2 | D3 | D4 | D5 | D6 | D7 | D8
---|---|---|---|---|---|---|---|---
Before starting | 8 | 3 | 5 | 2 | 1 | 2 | 1 | 1
D5 | 7 | 3 | 4 | 2 | 0 | 2 | 1 | 1
D8 | 6 | 2 | 3 | 1 | 0 | 1 | 1 | 0
D4 | 5 | 1 | 3 | 0 | 0 | 1 | 1 | 0
D2 | 4 | 0 | 3 | 0 | 0 | 1 | 1 | 0
D7 | 3 | 0 | 2 | 0 | 0 | 1 | 0 | 0
D6 | 2 | 0 | 1 | 0 | 0 | 0 | 0 | 0
D3 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0
D1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Table 4: An example of CAIML

##  5 Discussion

Assessing a person’s understanding of conceptual knowledge is not easy; as an
experimental model aimed to accomplish this, a great deal of thought and
research are required to realize its potential.

###  5.1 Dealing with similar Knowledge Points

There is a problem of how to deal with Knowledge Points that are different but
have little distinction, such as “random variation” and “random variable”. One
solution is to homogenize them to the same Knowledge Point; another is to
compensate one Knowledge Point’s Familiarity Measure with others’ Familiarity
Measures, because learning others helps to understand it. The compensation can
be calculated with Equation 10. $F_{k_{i}}$ is Knowledge Point $k_{i}$’s
Familiarity Measure, each of its sibling contributes $1/c_{j}$ of its
Familiarity Measure to $k_{i}$ ($c_{j}$ is coefficient to be determined).

$F_{k_{i}\\_new}=F_{k_{i}\\_old}+\sum_{j=1}^{m}{\frac{1}{c_{j}}}F_{k_{j}}$
(10)

###  5.2 Trade-offs of evaluating one’s understanding of conceptual knowledge

Quantitatively assessing one’s understanding of conceptual knowledge seems to
be a good thing, but there are risks that it introduces some harmful effects.
For example, if the Familiarity Measures and understanding degrees calculated
are inaccurate, it may lead to wrong decisions. In addition, it cannot detect
a person’s talent and potential in a field. On the other hand, traditional
exams or interviews have their limitations. For example, it needs other
people’s cooperation to accomplish the evaluation; it only assesses one’s
knowledge in a particular field at a time, and the evaluation is not
comprehensive. Since it only assesses questions being asked, not all of the
topics in a field are evaluated. ICKEM assesses one’s knowledge independently,
comprehensively, and automatically. Therefore, the methods of evaluating one’s
understanding of knowledge should be used cooperatively, complementing one
another.

###  5.3 Privacy issues

Recording one’s learning history will inevitably violates his privacy. To
protect privacy, the learning histories can be password protected or encrypted
and stored in personal storage; they should not be revealed to other people.
The only information that can be viewed by the outside world is the
individual’s knowledge measures of some Knowledge Points. The Knowledge Points
that may involve privacy are separated from others; every output of them
should be authorized by the owner. A person may choose to keep all of his
knowledge measures private.

###  5.4 The success criteria for ICKEM

For ICKEM to be a successful model of evaluating a person’s understanding of
conceptual knowledge, it should have a good performance of discovering a
person’s domain-level expertise and topic-level expertise. To evaluate its
performance, firstly, a group of subjects’ learning histories over a long time
(several years or decades of years) should be recorded.

For evaluating its ability of discovering domain-level expertise, each
subject’s domain-level expertise, such as Mathematics, Chemistry, Computer
Science, Biology etc., is determined through traditional methods like tests
and interviews. For each domain, a set of knowledge topics that are
distinctive for the domain is selected. Their Familiarity Measures are
calculated based on the subjects’ learning histories. Then randomly select a
number of knowledge topics from each domain and calculate the average
Familiarity Measure of them. The domain which has the highest average
Familiarity Measure is a subject’s domain-level expertise. Finally, compare
the domain-level expertise discovered by ICKEM with the expertise discovered
by traditional methods. If they are equivalent, it proves ICKEM’s capability
of discovering domain-level expertise.

For evaluating ICKEM’s ability of discovering topic-level expertise, randomly
select a set of knowledge topics from each domain, then evaluate a subject’s
understanding degrees about them, classifying them as “Understood” and “Not
Understood” according to the method mentioned in Section 3.1, then use
traditional methods to evaluate a subject’s understanding degrees about them,
classifying them as “Understood” and “Not Understood” too. By comparing the
results of both classification methods, we can obtain the precision and recall
of ICKEM’s capability of discovering topic-level expertise.

##  6 Related work

Many research fields focus on the collection of personal information, such as
lifelogging, expertise finding, and personal informatics. Bush envisioned the
‘memex’ system, in which individuals could compress and store personally
experienced information, such as books, records, and communications [8].
Inspired by ‘memex’, Gemmell et al. developed a project called MyLifeBits to
store all of a person’s digital media, including documents, images, audio, and
video [17]. In [31], a person’s reading history about an electronic document
is used as attributes for re-finding the document. ICKEM is similar to ‘memex’
and MyLifeBits in that it records an individual’s digital history, although
for a different purpose. ‘Memex’ and MyLifeBits are mainly for re-finding or
reviewing personal data; ICKEM is for quantitatively evaluating a person’s
knowledge.

Personal informatics is a class of tools that help people collect personally
relevant information for the purpose of self-reflection and gaining self-
knowledge [28, 55, 57]. Various tools have been developed to help people
collect and analyze different kinds of personal information, such as location
[30], finances [24], food [12], weight [23, 33], and physical activity [16].
ICKEM facilitates a new type of personal informatics tool that helps people
discover their expertise and deficiencies in a more accurate way, by
quantitatively assessing an individual’s understanding of knowledge.

Expertise is one’s expert skill or knowledge in a particular field. Expertise
finding is the use of tools for finding and assessing individual expertise
[38, 34, 54]. As an important link of knowledge sharing, expertise finding has
been heavily studied in many research communities [1, 10, 35, 51, 2, 18]. Many
sources of data have been exploited to assess an individual’s expertise, such
as one’s publications, documents, emails, web search behavior, other people’s
recommendations, social media etc. ICKEM provides a new source of data to
analyze one’s expertise–one’s learning history about a topic, which is more
comprehensive and straightforward than other data sources. Because one’s
expertise is mainly obtained through learning (Including “Informal Learning”,
which occurs through the experience of day-to-day situations, such as a casual
conversation, play, exploring, etc.)

##  7 Conclusion

People’s pursuing of knowledge is never stopping. Most conceptual knowledge is
transmitted through language; it is hard to imagine how a person can obtain
conceptual knowledge without using language. A piece of written or spoken
language can be converted into text. We propose a new method to estimate a
person’s understanding of a piece of conceptual knowledge, by analyzing the
text content of one’s all learning experiences about a knowledge topic. The
computation of familiarity degree takes into account the total time the
subject has devoted to a knowledge topic, a topic’s share in a learning
session, the subject’s physical and psychological status during a session, the
memory decay of each learning experience over time, and the difference among
learning methods. To estimate a person’s understanding degree to a knowledge
topic, it comprehensively evaluates one’s familiarity degrees to the topic
itself and other topics that are essential to understand the topic.
Quantitatively evaluating a person’s understanding of knowledge facilitates
many applications, such as personalized recommendation, meaningful learning,
expertise and deficiency finding etc. With the prevailing of wearable
computers like Google Glass and Apple Watch, and maturing of technologies like
Speech Recognition and Optical Character Recognition (OCR), it is practicable
to analysis people’s daily learning activities like talking, listening, and
reading. Therefore, ICKEM is technically feasible and beneficial.

## References

  * [1] M. S. Ackerman, J. Dachtera, V. Pipek, and V. Wulf.  Sharing knowledge and expertise: The cscw view of knowledge management.  Computer Supported Cooperative Work (CSCW), 22(4-6):531–573, 2013\.
  * [2] Ç. Aslay, N. O’Hare, L. M. Aiello, and A. Jaimes.  Competition-based networks for expert finding.  In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pages 1033–1036. ACM, 2013\.
  * [3] L. Averell and A. Heathcote.  The form of the forgetting curve and the fate of memories.  Journal of Mathematical Psychology, 55(1):25–35, 2011.
  * [4] V. A. Benassi, C. Overson, and C. M. Hakala.  Applying science of learning in education: Infusing psychological science into the curriculum.  2014\.
  * [5] D. M. Blei.  Probabilistic topic models.  Communications of the ACM, 55(4):77–84, 2012.
  * [6] D. M. Blei, A. Y. Ng, and M. I. Jordan.  Latent dirichlet allocation.  the Journal of machine Learning research, 3:993–1022, 2003.
  * [7] J. D. Bransford, A. L. Brown, and R. R. Cocking.  How people learn: Brain, mind, experience, and school. National Academy Press, 1999.
  * [8] V. Bush.  As we may think.  ACM SIGPC Notes, 1(4):36–44, 1979.
  * [9] T. Cheng, G. C. Migliaccio, J. Teizer, and U. C. Gatti.  Data fusion of real-time location sensing and physiological status monitoring for ergonomics analysis of construction workers.  Journal of Computing in Civil engineering, 27(3):320–335, 2012\.
  * [10] Z. Cheng, J. Caverlee, H. Barthwal, and V. Bachani.  Who is the barbecue king of texas?: a geo-spatial approach to finding local experts on twitter.  In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 335–344. ACM, 2014.
  * [11] B. Á. Cliath, O. D. F. Rialtais, T. S. Alliance, S. T. Laighean, F. Rialtais, and A. R. Post-tráchta.  Learning for life: White paper on adult education, 2000.
  * [12] F. Cordeiro, D. A. Epstein, E. Thomaz, E. Bales, A. K. Jagannathan, G. D. Abowd, and J. Fogarty.  Barriers and negative nudges: Exploring challenges in food journaling.  In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 1159–1162. ACM, 2015.
  * [13] M. J. DeRemer, R. B. Streeter, C. D. Mielcarz, G. S. Lewis, and D. McDonald.  Physiological status monitoring system, May 12 2015.  US Patent 9,028,404.
  * [14] H. Ebbinghaus.  Memory: A contribution to experimental psychology.  Number 3. University Microfilms, 1913.
  * [15] E. O. Finkenbinder.  The curve of forgetting.  The American Journal of Psychology, 24(1):8–32, 1913.
  * [16] T. Fritz, E. M. Huang, G. C. Murphy, and T. Zimmermann.  Persuasive technology in the real world: a study of long-term use of activity sensing devices for fitness.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 487–496. ACM, 2014.
  * [17] J. Gemmell, G. Bell, R. Lueder, S. Drucker, and C. Wong.  Mylifebits: fulfilling the memex vision.  In Proceedings of the tenth ACM international conference on Multimedia, pages 235–238. ACM, 2002.
  * [18] I. Guy, U. Avraham, D. Carmel, S. Ur, M. Jacovi, and I. Ronen.  Mining expertise and interests from social media.  In Proceedings of the 22nd international conference on World Wide Web, pages 515–526. ACM, 2013.
  * [19] O. Heller, W. Mack, and J. Seitz.  Replikation der ebbinghaus’ schen vergessenskurve mit der ersparnismethode:“das behalten und vergessen als funktion der zeit”.  Zeitschrift für Psychologie mit Zeitschrift für angewandte Psychologie, 199(1):3–18, 1991.
  * [20] J. Hiebert.  Conceptual and procedural knowledge: The case of mathematics.  Routledge, 2013.
  * [21] T. Hofmann.  Probabilistic latent semantic indexing.  In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 50–57. ACM, 1999.
  * [22] D. P. Hunt.  The concept of knowledge and how to measure it.  Journal of intellectual capital, 4(1):100–113, 2003.
  * [23] M. Kay, D. Morris, J. A. Kientz, et al.  There’s no such thing as gaining a pound: Reconsidering the bathroom scale user interface.  In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 401–410. ACM, 2013.
  * [24] J. J. Kaye, M. McCuistion, R. Gulotta, and D. A. Shamma.  Money talks: tracking personal finances.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 521–530. ACM, 2014.
  * [25] H. Kurniawan, A. V. Maslov, and M. Pechenizkiy.  Stress detection from speech and galvanic skin response signals.  In Computer-Based Medical Systems (CBMS), 2013 IEEE 26th International Symposium on, pages 209–214. IEEE, 2013.
  * [26] J. Lalley and R. Miller.  The learning pyramid: Does it point teachers in the right direction.  Education, 128(1):16, 2007.
  * [27] K. Letrud.  A rebuttal of ntl institute’s learning pyramid.  Education, 133(1):117–124, 2012.
  * [28] I. Li, A. Dey, and J. Forlizzi.  A stage-based model of personal informatics systems.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 557–566. ACM, 2010.
  * [29] H. Lin, J. Jia, Q. Guo, Y. Xue, Q. Li, J. Huang, L. Cai, and L. Feng.  User-level psychological stress detection from social media using deep neural network.  In Proceedings of the 22nd ACM international conference on Multimedia, pages 507–516. ACM, 2014.
  * [30] J. Lindqvist, J. Cranshaw, J. Wiese, J. Hong, and J. Zimmerman.  I’m the mayor of my house: examining why people use foursquare-a social-driven location sharing application.  In Proceedings of the SIGCHI conference on human factors in computing systems, pages 2409–2418. ACM, 2011.
  * [31] G. Liu, B. Jiang, and L. Feng.  A PDF document re-finding system with a q&a wizard interface.  Knowl.-Based Syst., 131:1–9, 2017.
  * [32] S. Magennis and A. Farrell.  Teaching and learning activities: Expanding the repertoire to support student learning.  Emerging issues in the practice of university learning and teaching, 1, 2005.
  * [33] J. Maitland and M. Chalmers.  Designing for peer involvement in weight management.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 315–324. ACM, 2011.
  * [34] D. Mattox, M. T. Maybury, and D. Morey.  Enterprise expert and knowledge discovery.  In Proceedings of the 8th International Conference on Human-Computer Interactions, pages 303–307, 1999.
  * [35] M. Maybury, R. D’Amore, and D. House.  Awareness of organizational expertise.  International Journal of Human-Computer Interaction, 14(2):199–217, 2002.
  * [36] R. E. Mayer.  Rote versus meaningful learning.  Theory into practice, 41(4):226–232, 2002.
  * [37] R. McCormick.  Conceptual and procedural knowledge.  International journal of technology and design education, 7(1-2):141–159, 1997.
  * [38] D. W. McDonald and M. S. Ackerman.  Just talk to me: a field study of expertise location.  In Proceedings of the 1998 ACM conference on Computer supported cooperative work, pages 315–324. ACM, 1998.
  * [39] J. M. Murre and J. Dros.  Replication and analysis of ebbinghaus’ forgetting curve.  PloS one, 10(7):e0120644, 2015.
  * [40] B. Ni, G. Wang, and P. Moulin.  Rgbd-hudaact: A color-depth video database for human daily activity recognition.  In A. Fossati, J. Gall, H. Grabner, X. Ren, and K. Konolige, editors, Consumer Depth Cameras for Computer Vision, Research Topics and Applications, Advances in Computer Vision and Pattern Recognition, pages 193–208. Springer, 2013.
  * [41] C. H. Papadimitriou, H. Tamaki, P. Raghavan, and S. Vempala.  Latent semantic indexing: A probabilistic analysis.  In Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems, pages 159–168. ACM, 1998.
  * [42] R. Paradise and B. Rogoff.  Side by side: Learning by observing and pitching in.  Ethos, 37(1):102–138, 2009.
  * [43] O. PISA.  Measuring student knowledge and skills.  The PISA Assessment of Reading. Mathematical and Scientific Literacy, 2000.
  * [44] L. Piyathilaka and S. Kodagoda.  Gaussian mixture based hmm for human daily activity recognition using 3d skeleton features.  In Industrial Electronics and Applications (ICIEA), 2013 8th IEEE Conference on, pages 567–572. IEEE, 2013.
  * [45] D. D. Qian and M. Schedl.  Evaluation of an in-depth vocabulary knowledge measure for assessing reading performance.  Language Testing, 21(1):28–52, 2004.
  * [46] D. C. Rubin and A. E. Wenzel.  One hundred years of forgetting: A quantitative description of retention.  Psychological review, 103(4):734, 1996.
  * [47] D. L. Schacter.  Psychology, Second Edition.  Worth Publishers, 2011.
  * [48] D. A. Sousa.  How the brain learns: A classroom teacher’s guide.  ERIC, 2001.
  * [49] M. Steyvers and T. Griffiths.  Probabilistic topic models.  Handbook of latent semantic analysis, 427(7):424–440, 2007.
  * [50] J. Sung, C. Ponce, B. Selman, and A. Saxena.  Unstructured human activity detection from rgbd images.  In Robotics and Automation (ICRA), 2012 IEEE International Conference on, pages 842–849. IEEE, 2012.
  * [51] J. Tang, J. Zhang, L. Yao, J. Li, L. Zhang, and Z. Su.  Arnetminer: extraction and mining of academic social networks.  In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 990–998. ACM, 2008.
  * [52] W. S. Terry.  Learning and memory: Basic principles, processes, and procedures.  Psychology Press, 2015.
  * [53] A. Tough.  The adult’s learning projects. a fresh approach to theory and practice in adult learning.  1979\.
  * [54] A. S. Vivacqua.  Agents for expertise location.  In Proc. 1999 AAAI Spring Symposium Workshop on Intelligent Agents in Cyberspace, pages 9–13, 1999.
  * [55] G. Wolf.  Know thyself: Tracking every facet of life, from sleep to mood to pain.  Wired Magazine, 365, 2009.
  * [56] C.-M. Yang.  Mental and physical health status monitoring, analyze and automatic follow up methods and its application on clothing, Feb. 14 2005.  US Patent App. 11/056,227.
  * [57] N. Yau and J. Schneider.  Self-surveillance.  Bulletin of the American Society for Information Science and Technology, 35(5):24–30, 2009.