11institutetext: Research Studios Austria FG, Vienna 1090, Austria
22institutetext: Vienna University of Technology (TU Wien), Vienna 1040,
Austria 22email<EMAIL_ADDRESS>
# Relevance Models Based on the Knowledge Gap

Yasin Ghafourian 1122 0000-0001-9683-9748

###### Abstract

Search systems are increasingly used for gaining knowledge through accessing
relevant resources from a vast volume of content. However, search systems
provide only limited support to users in knowledge acquisition contexts.
Specifically, they do not fully consider the knowledge gap which we define as
the gap existing between what the user knows and what the user intends to
learn. The effects of considering the knowledge gap for knowledge acquisition
tasks remain largely unexplored in search systems. We propose to model and
incorporate the knowledge gap into search algorithms. We plan to explore to
what extent the incorporation of the knowledge gap leads to an improvement in
the performance of search systems in knowledge acquisition tasks. Furthermore,
we aim to investigate and design a metric for the evaluation of the search
systems’ performance in the context of knowledge acquisition tasks.

###### Keywords:

Information Retrieval Knowledge Delta Knowledge Acquisition.

##  1 Motivation and Problem Statement

Web search is nowadays considered more to be a source for accessing
information resources and exploration, and educational resources are also not
an exception to this. Web search has become the most popular medium for
education in schools, universities and for professional training [1, 2]. In
addition, web search is being used more often for the aim of gaining new
knowledge [3]. There have been numerous studies done on the information-
seeking behaviors of students and academic staff in different parts of the
world (some examples are [4, 5, 6, 7]) which also verify that the majority of
individuals and students consider the internet to be the most useful
information source for learning.

Marchionini [2] categorises the search activities into two broad categories of
_look-up_ and _exploratory_ search activities. Look up search is the activity
in which users know in particular what information they want and have a
concrete expectation of what would the desired search results be, while in
exploratory search users go through multiple iterations of searching the
online resources. With regard to using search systems for learning purposes,
look-up search is viewed as a type of search that users initiate based on
their current knowledge of the subject of interest that leads them to the
relevant neighborhood of information. From this neighborhood, users will then
start their exploration into the learning resources and might reformulate
their query multiple times according to their findings with the aim of
reaching more satisfying resources.

Users typically have different levels of background knowledge on the topic in
question. We define a user’s background knowledge as the current level of the
user’s familiarity with the topic. Due to different levels of background
knowledge, users might perceive the relevance of the documents on a topic
differently.

Search engines currently do not take this diversity in the background
knowledge levels of the users into account and assume that users’ information
needs are well represented in their queries [8]. This is a clear limitation.
The optimal sequence of documents leading to satisfying a user’s learning goal
depends on the user’s specific background knowledge. What if search engines
could exploit information about the user’s background knowledge to provide the
suitable (sequence of) documents helping the user to satisfy their learning
goal as fast as possible.

As search engines are being used by students and researchers, it is crucial
for search engines to pursue more developments in this direction so as to be a
better fit for learning tasks, especially _knowledge acquisition tasks_. We
define _knowledge acquisition tasks_ as tasks in which users aim to acquire
knowledge with a learning goal as part of a sequence of interactions with an
information retrieval (IR) system. Supposing that we have a search system
specialized for knowledge acquisition tasks that takes the users’ knowledge
level into account, it will provide the users with resources that best fit
their learning needs according to their knowledge level. This, in turn, saves
effort from users in terms of spending longer search times and consuming
documents that are returned as relevant but cannot be utilised by the users
according to their knowledge level.

To incorporate the users’ knowledge level into the search, one needs to enable
the search system to have a model representing the users’ knowledge within the
topic of interest (“what the user knows”). Furthermore, one needs to define
and represent the knowledge to be acquired (“what the user wants to know”).
The goal is then to overcome the gap between these two representations. We
call this gap the _knowledge gap_. Having defined the knowledge gap, the
objective is to develop a retrieval method that provides users with an order
list (a path) of resources, that helps them to overcome the knowledge gap. A
suitable order will be one where more complex resources requiring more
background knowledge will be preceded by resources that are more easily
approachable based on the user’s background knowledge. In this research, we
will investigate the means to measure the _knowledge gap_ and understand how
one applies it within a search system designed for acquiring knowledge so that
the users can effectively overcome the knowledge gap. In the rest of this
paper, we will first discuss the research questions that we seek to answer
throughout this research. Later and in the section that follows, we will
provide a brief overview of the research surrounding the concepts of the
_knowledge gap_ and the _knowledge delta_. Finally, we will explain our
planned methodology to approach the research questions.

##  2 Research questions

Our motivation is to investigate on how to improve the retrieval effectiveness
of the search systems for knowledge acquisition by incorporating the knowledge
gap into the ranking mechanisms. We propose our main research question as
follows:

High level research question: How can the search system help users to
effectively overcome their knowledge gap in a knowledge acquisition task?

This high-level research question is comprised of three fine-grained research
questions to be investigated:

  * •

RQ1: How can we model users’ background knowledge, target knowledge and the
knowledge gap in knowledge acquisition tasks?

  * •

RQ2: To what extent can the incorporation of the knowledge gap into a search
system facilitate a more efficient journey for users in knowledge acquisition
tasks?

  * •

RQ3: How should search systems be evaluated in the context of knowledge
acquisition tasks?

##  3 Background and related work

In this section we provide the result of our literature review done in an
attempt to capture the viewpoint of the papers that discuss the concepts of
_knowledge gap_ and _knowledge delta_ and how they approach and incorporate
these concepts.

###  3.1 Knowledge Gap

What we defined earlier, i.e the gap between the users’ knowledge level and
the level of knowledge in the field of interest for learning has been
discussed in the literature under the title of _knowledge gap_. Knowledge gap
is one of the causing factors of information need [9]. Another factor that
causes information need is referred to as _Anomaly in the state of knowledge_
by Belkin et al. [10] which is the phenomenon in the state of knowledge that
causes the information need. In one of the early studies on information use,
Dervin and Nilan [11] used the phrase _knowledge gap_ to refer to a situation
where a person’s cognitive state has recognized an incompleteness in its
currently possessed information. This incompleteness happens as a result of
interaction with information sources or through thinking processes, and
thereby later that incompleteness will turn into an information need.
Additionally, Thellefsen et al. [12] use _knowledge gap_ and _information
need_ interchangeably in a discussion where some of the definitions of
_knowledge gap_ are covered to argue the intricacy of the concept of
information need and the importance of incorporating users’ information need
while developing a knowledge organization system. In a research done by Yu et
al. [13] the _knowledge gap_ also has a similar explanation, however, the
research is more focused on knowledge predictive models for the knowledge
state of users which are being calibrated through questionnaires. Considering
the definition of _knowledge gap_ , there are studies that use the same
definition and provide search solutions that are adaptive to the _knowledge
gap_ [14, 15, 16]. In addition, there are similar studies that seek to model
the knowledge gap by modeling the knowledge of the user and compare it against
an existing knowledge level for the topic of interest [1, 17, 18, 19, 20, 21].
Among these works, the work done by Zhao et al. [21] explores the knowledge
paths between users’ already acquired knowledge and the target knowledge that
the user aims to obtain. Considering this knowledge path and in the context of
recommender systems, the authors’ method recommends a number of papers to the
users so that their learning goals are achieved in the best satisfying way.
Similarly, as our goal will be to assist users in knowledge acquisition tasks,
we will take into consideration the target knowledge that users aim to obtain.
Having modeled the knowledge from users’ background knowledge and the target
knowledge level, our goal is to estimate the knowledge gap between these
levels. Thereafter, we will use the modeled knowledge gap to improve the
retrieval effectiveness of the search systems for knowledge acquisition.

###  3.2 Knowledge Delta

_Knowledge delta_ is another concept that is semantically closely related to
knowledge gap. One definition of _knowledge delta_ in the literature is the
amount of knowledge change in a user’s knowledge level which can be measured
through questionnaires. As a case in point, we can refer to the work of
Grunewald et al. [22], where the expertise gain is calculated through asking
users of a ”Massive Open Online Courses” system about their knowledge level in
a field before and after taking an online course and denoting it as _knowledge
delta_. Similar studies have also been done to measure the knowledge change of
the users [22, 23, 24, 25, 26].

In all the aforementioned works, _knowledge delta_ is used as a concept that
demonstrates the change in users knowledge level. This concept is also
associated with the name of _knowledge gain_ in other works such as [27].

##  4 Research Methodology

The methodology begins with the investigation to find an answer to the first
research question all the way to the third research question. The steps of the
methodology are three-fold. Firstly, we will build a model for a user’s
knowledge and use it to build a model for the user’s knowledge gap. Secondly,
we will ask users to participate in knowledge acquisition tasks to gain
knowledge on learning goals that will be defined for them. During this
experiment, we will utilize the modeled knowledge gap for each user during the
user’s search session in order to provide the user with better results for
learning. during this step, we will investigate the extent of improvement that
incorporating the knowledge gap will bring about in a learning session.
Thirdly, we will design a function whose output is a measure that will score
the user’s learning progress throughout the session.

RQ1: How can we model users’ background knowledge, target knowledge and the
knowledge gap in knowledge acquisition tasks?

To answer this research question, we need to design learning tasks for the
users and we need to estimate the users’ domain knowledge. Each learning task
will set gaining knowledge on a sub-topic within a topic as a goal for the
users. This goal will establish a desired level of knowledge as the target
knowledge that a user wants to acquire. We will choose a fixed number of sub-
topics and assign each user to a sub-topic for the learning task.

As it was mentioned under section 3, several studies have modeled a user’s
knowledge in a variety of tasks and in different ways [1, 16, 21, 28, 29].
Extending the work done by Zhang et al [29], we aim to represent the user’s
knowledge of a topic with a set of concepts within that topic .

We will assess the user’s knowledge before and after the learning session
using knowledge tests. We will design the knowledge tests in such a way that
the user’s knowledge of each concept can be attributed to a level of learning
according to Bloom’s taxonomy of educational objectives [30, 31].

Having completed the knowledge test, each user’s knowledge will be represented
with its set of constituent concepts. There are a variety of options to
represent the user’s knowledge. One such way is a one-dimensional vector of
concepts and the user’s understanding of the concepts as weights. The same
model will also be used to represent the target knowledge. The knowledge gap
will then be computed between these two models.

Having modeled the knowledge gap, the methodology continues the investigation
by moving to the second research question.

RQ2: To what extent can the incorporation of the knowledge gap into a search
system facilitate a more efficient journey for users in knowledge acquisition
tasks?

After the pre-task knowledge assessment, users will have access to a search
interface connected to a search engine (here after collectively referred to as
the search system) to carry out their learning tasks. The interface will allow
for the recording of the users’ interactions which will later be used as
features to evaluate the quality of the learning sessions.

Each user will participate in at least two learning sessions during this
experiment. In one learning session, the search system will not take the
knowledge gap into account and original retrieved results by the system will
be presented to the user. In the other learning session, the search system
will take the knowledge gap into account and adapt the results before
presenting them to the user. There are several ways to adapt the search
results to the knowledge gap. One such way will be to re-rank the results of
the user’s each query submission based on a personalized understandability
score of those results. For each of the learning sessions, we will design a
different learning task. So as to ensure that learning about one topic doesn’t
affect the user’s knowledge about other topics. After the learning session,
the users will be asked to take a knowledge test again (Post-task knowledge
tests). Comparing the result of the pre-task knowledge test with the post-task
test for each user, we will have a score for the progress of the user’s
knowledge. On the other hand, for each user, we will compare the knowledge
gained in the learning sessions to observe the effect of incorporating the
knowledge gap on the quality of the learning session. We will evaluate this
quality based on the interaction features recorded during the session.

RQ3: How should search systems be evaluated in the context of knowledge
acquisition tasks?

In this research question, we will investigate how to define evaluation
performance metrics for search systems in knowledge acquisition tasks.
Previously in [32], we have explained why current retrieval metrics are
insufficient in this context as they treat each query during the session
independently and don’t consider the users’ knowledge factor in the
evaluations. Correspondingly, we proposed three directions forward as well as
their advantages and shortcomings: 1) online evaluation approach, 2)
prerequisite-labeled relevance judgements approach, and 3) session-based
evaluation approach. It’s essential to mention at this point that what has
been defined in this research proposal as _knowledge gap_ was defined as
_knowledge delta_ in [32]. However, in order to maintain a greater consistency
with the literature in this area considering the subtle distinction between
the _knowledge delta_ and the _knowledge gap_ , we have adopted the term
_knowledge gap_ in this research proposal.

Our objective will be to extend this previous work and formalise the _session-
based evaluation approach_. The goal is to design a session-based evaluation
function that gauges the quality of learning sessions in knowledge acquisition
tasks.

Up until this point in the methodology, we will have collected data on
knowledge that the users have gained and information about learning sessions.
In addition we will have recorded the users’ interactions with the search
system. As a result, a function that will serve as a new performance metric
will be designed. This performance metric will use interaction features, such
as time, number of queries used, etc., as cost indicators. It will combine
these cost indicators such that the function’s value aligns well with the
experiences from the qualitative data (Knowledge tests). As a result, for the
evaluation of future learning sessions, it will suffice to only have access to
cost indicators and to use the designed function to evaluate the learning
session.

There two main research challenges for the implementation of the discussed
methodology:

  1. 1.

The challenge that exists for the implementation for research question two is
maintaining a balance between guiding a user to resources that are better
suited for them and are adapted to their level of knowledge by the system, and
just responding to the queries the user submitted.

  2. 2.

In defining the learning goal for the experiments, the level of understanding
of the user in the topic is a variable and the desired change in the level of
understanding for the sake of experiments should be fixed. (e.g. Does it
suffice if users are just familiarized with a concept, or should they be able
to use it or should they be able to implement it?)

## Acknowledgements

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under the Marie Skłodowska-Curie grant
agreement No 860721.

## References

  * [1] Lora Aroyo, Ronald Denaux, Vania Dimitrova and Michael Pye  “Interactive ontology-based user knowledge acquisition: A case study”  In _European Semantic Web Conference_ , 2006, pp. 560–574  Springer
  * [2] Gary Marchionini  “Exploratory search: from finding to understanding”  In _Communications of the ACM_ 49.4 ACM New York, NY, USA, 2006, pp. 41–46
  * [3] Ujwal Gadiraju, Ran Yu, Stefan Dietze and Peter Holtz  “Analyzing knowledge gain of users in informational search sessions on the web”  In _Proceedings of the 2018 Conference on Human Information Interaction & Retrieval_, 2018, pp. 2–11
  * [4] al-Haddabi Huda Sulaiman  “Information needs and information seeking behaviour at the College of Medicine, The Sultan Qaboos University, Oman/Huda Sulaiman al-Haddabi”, 2005
  * [5] Ntando Nkomo  “A comparative analysis of the web information seeking behaviour of students and staff at the University of Zululand and the Durban University of Technology”, 2009
  * [6] MAGNUS O IGBINOVIA and IGUEH IJ CLN IKENWE  “Information Seeking Behavior of Academic Librarians for Effective Performance: A study of UNIBEN, AAU and AUCHI Poltechnic, EDO State, Nigeria.”, 2014
  * [7] Kwesi Gyesi  “Information Seeking Behaviour of Graduate Students of the University of Professional Studies, Accra (UPSA)”  In _Library Philosophy and Practice (e-journal)_ 4155, 2020
  * [8] Diane Kelly and Nicholas J Belkin  “A user modeling system for personalized interaction and tailored retrieval in interactive IR”  In _Proceedings of the American Society for Information Science and Technology_ 39.1 Wiley Online Library, 2002, pp. 316–325
  * [9] Erica Cosijn and Peter Ingwersen  “Dimensions of relevance”  In _Information Processing & Management_ 36.4 Elsevier, 2000, pp. 533–550
  * [10] Nicholas J Belkin, Robert N Oddy and Helen M Brooks  “ASK for information retrieval: Part I. Background and theory”  In _Journal of documentation_ MCB UP Ltd, 1982
  * [11] Brenda Dervin and Michael Nilan  “Information needs and uses”  In _M. Williams (Ed.), Annual Review of Information Science and Technology_ 21, 1986, pp. 3–33  White Plains, NY: Knowledge Industry Publications
  * [12] Martin Thellefsen, Torkild Thellefsen and Brent Sørenson  “A pragmatic semeiotic perspective on the concept of information need and its relevance for knowledge organization”  In _KO KNOWLEDGE ORGANIZATION_ 40.4 Nomos Verlagsgesellschaft mbH & Co. KG, 2014, pp. 213–224
  * [13] Ran Yu et al.  “Topic-independent modeling of user knowledge in informational search sessions”  In _Information Retrieval Journal_ 24.3 Springer, 2021, pp. 240–268
  * [14] Nenad Stojanovic  “On the role of a user’s knowledge gap in an information retrieval process”  In _Proceedings of the 3rd international conference on Knowledge capture_ , 2005, pp. 83–90
  * [15] Luca Soldaini  “The knowledge and language gap in medical information seeking”  In _ACM SIGIR Forum_ 52.2, 2019, pp. 178–179  ACM New York, NY, USA
  * [16] Yao Zhang and Chang Liu  “Users’ Knowledge Use and Change during Information Searching Process: A Perspective of Vocabulary Usage”  In _Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020_ , 2020, pp. 47–56
  * [17] Ye Xiong  “An Automated Feedback System to Support Student Learning of Conceptual Knowledge in Writing-To-Learn Activities”, 2020
  * [18] Khushboo Thaker, Lei Zhang, Daqing He and Peter Brusilovsky  “Recommending Remedial Readings Using Student Knowledge State.”  In _International Educational Data Mining Society_ ERIC, 2020
  * [19] Stefanie N Lindstaedt, Günter Beham, Barbara Kump and Tobias Ley  “Getting to know your user–unobtrusive user model maintenance within work-integrated learning environments”  In _European Conference on Technology Enhanced Learning_ , 2009, pp. 73–87  Springer
  * [20] Rohail Syed  “Models and Algorithms for Understanding and Supporting Learning Goals in Information Retrieval”, 2020
  * [21] Weidong Zhao, Ran Wu and Haitao Liu  “Paper recommendation based on the knowledge gap between a researcher’s background knowledge and research target”  In _Information processing & management_ 52.5 Elsevier, 2016, pp. 976–988
  * [22] Franka Grünewald, Christoph Meinel, Michael Totschnig and Christian Willems  “Designing MOOCs for the support of multiple learning styles”  In _European conference on technology enhanced learning_ , 2013, pp. 371–382  Springer
  * [23] M Monica Daghio, Giuseppe Fattori and Anna V Ciardullo  “Assessment of readability and learning of easy-to-read educational health materials designed and written with the help of citizens by means of two non-alternative methods”  In _Advances in health sciences education_ 11.2 Springer, 2006, pp. 123–132
  * [24] M Monica Daghio, Giuseppe Fattori and Anna V Ciardullo  “Evaluation of easy-to-read information material on healthy life-styles written with the help of citizens’ collaboration through networking”  In _Promotion & education_ 13.3 SAGE Publications SAGE UK: London, England, 2006, pp. 191–196
  * [25] A Schaumberg  “Variation in closeness to reality of standardized resuscitation scenarios: Effects on the success of cognitive learning of medical students”  In _Der Anaesthesist_ 64.4, 2015, pp. 286–291
  * [26] Henrique Carvalho Resende, Nina Slamnik-Krijestorac, Cristiano Bonato Both and Johann Marquez-Barja  “Introducing Engineering Undergraduate Students to Network Management Techniques: a Hands-on approach using the Citylab Smart City”  In _2020 IEEE Global Engineering Education Conference (EDUCON)_ , 2020, pp. 1316–1324  IEEE
  * [27] Pertti Vakkari et al.  “Modeling the usefulness of search results as measured by information use”  In _Information Processing & Management_ 56.3 Elsevier, 2019, pp. 879–894
  * [28] Ran Yu et al.  “Predicting user knowledge gain in informational search sessions”  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, 2018, pp. 75–84
  * [29] Xiangmin Zhang, Jingjing Liu, Michael Cole and Nicholas Belkin  “Predicting users’ domain knowledge in information retrieval using multiple regression analysis of search behaviors”  In _Journal of the Association for Information Science and Technology_ 66.5 Wiley Online Library, 2015, pp. 980–1000
  * [30] BS Bloom  “Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain. New York: David McKay Co Inc. as cited in file”  In _D:/bloom. html_ , 1956
  * [31] Lorin W Anderson and Benjamin Samuel Bloom  “A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives”  Longman,, 2001
  * [32] Yasin Ghafourian, Petr Knoth and Allan Hanbury  “Information retrieval evaluation in knowledge acquisition tasks”  In _Proceedings of the Third Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR)_ , 2021, pp. 88–95  Canberra, Australia, Virtual event