PSFinal Set of Selected Primary Studies

# Immersive Learning Frameworks:
A Systematic Literature Review

Filipe Arantes Fernandes, Claudia Susie Camargo Rodrigues, Eldânae Teixeira
and Cláudia Werner

###### Abstract

Contribution: This secondary study examines the literature on immersive
learning frameworks and reviews their state of the art. Frameworks have been
categorized according to their purpose. In addition, the elements that compose
them were also categorized. Some gaps were identified and proposed as a
research roadmap.

Background: Immersive technologies for education have been used for some
years. Despite this, there are few works that aim to support the development
and use of virtual environments for immersive learning.

Research Questions: This systematic review has the following main research
question: What is the state of the art of immersive learning frameworks? In
order to answer this question, secondary research questions were defined: 1)
what definitions of immersive learning were adopted in primary studies? 2)
what are the purposes of use by the frameworks? 3) what are the elements that
compose the frameworks? 4) what are the methods used to validate the
frameworks?

Methodology: As per the systematic review guidelines, this study followed a
rigorous and replicable process for collecting and analyzing data. From 1721
articles identified in the search engines, 15 were selected after the
inclusion and exclusion criteria.

Findings: Most frameworks are models that investigate the causal relationship
between immersive learning factors that influence learning outcomes. Although
this theoretical aspect is important for the advancement of research, the area
still lacks more practical frameworks that address more technical details and
support development, as well as the use of immersive virtual environments by
teachers and instructors.

###### Index Terms:

Augmented and virtual reality, education, educational virtual environment,
extended reality, framework, guideline, immersive education, immersive
learning, immersive virtual environment, model, systematic review.

††publicationid: pubid: 0000–0000/00$00.00 ©2022 IEEE

##  I Introduction

In October 2021, Facebook, now called
Meta111https://about.fb.com/news/2021/10/facebook-company-is-now-meta/,
announced the plan to create the metaverse, a kind of universe in Augmented
and Virtual Reality (AVR). After this announcement, the possibility of using
immersive technologies for business, marketing, games and education became
even more popular [narin2021content].

Immersive technologies for education have been used for several years, mainly
because virtual environments give the user the feeling of being present in the
context that is presented, in addition to allowing the virtual manipulation of
objects [greenwald2017technology, psotka1996immersive]. In situations where
being physically present would be too expensive, dangerous or impossible,
immersive experiences bring many advantages, for example in the training of
surgical skills, pilots and astronauts [psotka1996immersive]. Furthermore,
immersive experiences have greater engagement and allow greater interactivity
of the student with the instructional material, encourages the collaborative
construction of knowledge, presents more contextualized tasks, less abstract
instructions and favors reflective practice [bailenson2008use,
lee2014learning].

Although virtual environments have already evolved a lot, there are still many
research challenges involving immersive technologies in education [RW18, RW49,
ghinea2014mulsemedia]. In addition to the complexity of computational
solutions for the specifics of educational demands, generally developed by
researchers in the field of computing, there are challenges regarding
pedagogical and psychological aspects, as well as user experience,
storytelling, simulator sickness and others. In parallel, devices have evolved
rapidly, allowing the use of virtual environments both in traditional devices
(desktop and mobile) and in immersive devices (head-mounted display, motion
sensors, and others). Despite the range of devices, developing for the various
platforms is also challenging in order to ensure effective platform-
independent performance.

O State of XR Report resume os principais desafios para a adoção de
tecnologias imersivas na educação em 6 categorias: access, affordability,
inadequate XR teacher training programs, interoperability, lack of content and
lack of infrastructure and tech support.

A fim de solucionar os desafios mencionados acima e de contribuir com a
evolução das pesquisas relacionadas às tecnologias imersivas na educação,
Immersive Learning Frameworks têm sido uma estratégia que pesquisadores têm
encontrado de definir uma estrutura conceitual básica a fim de reunir
conceitos e conceber uma compreensão abrangente de um determinado fenômeno no
contexto de immersive learning [regoniel2015conceptual, jabareen2009building].

Portanto, considerando os desafios das tecnologias imersivas na educação e
frameworks como soluções a estes desafios, esta revisão sistemática da
literatura tem como objetivo fornecer evidências sobre o estado da arte de
Immersive Learning Frameworks. Mais precisamente, estamos interessados em
entender como os frameworks contribuem para a solução dos principais desafios,
segundo o State of XR Report, além de identificar lacunas e oportunidades para
pesquisas futuras.

This article is organized as follows: Section II defines mains concepts used
throughout this work. Section III presents some previous secondary studies on
Immersive Learning. Section IV describes the research method and the article
selection process. Section V presents the answers to the research questions.
Section VI discusses the relevant findings and, finally, conclusions and
future work are presented in Section VII.

##  II Background

O objetivo desta seção é apresentar os principais conceitos envolvidos neste
estudo secundário.

####  II-1 VR, MR and XR

Milgram and Kishino [milgram1994taxonomy] definiram a transição entre
realidade e virtualidade por meio de um diagrama conhecido como Milgram’s
continuum, conforme apresentada na Fig.1. Em uma extremidade está o ambiente
real e na outra o ambiente virtual, sendo que no intervalo, estão a Realidade
Aumentada (AR) e a Virtualidade Aumentada (AV). AR é a inserção de elementos
virtuais no ambiente real e AV, por sua vez, é a incorporação de elementos
reais ao ambiente virtual. Tanto AR quanto a AV são partes de uma realidade
mais ampla, chamada de Realidade Misturada (MR). Sendo assim, tecnologia de MR
possui a capacidade de misturar elementos do mundo real e do virtual. Um termo
abrangente que engloba todo o espectro da virtualidade contínua é denominada
eXtended Reality (XR). Uma das principais características da XR é de imergir o
usuário em um ambiente totalmente sintético gerado por computador. Desta
forma, devido à característica de imersão, a literatura aborda tanto o termo
XR quanto “imersivo” aos sistemas computacionais compreendidos na Virtuality
Continuum.

Figure 1: Virtuality continuum [milgram1994taxonomy].

####  II-2 Immersive Technologies

Remete aos sistemas computacionais (hardware e software) que possibilitam uma
interface humano-computador mais intuitiva por meio de dispositivos e sensores
que interagem com até os 5 sentidos humanos. Os principais dispositivos são
Head-Mounted Display (HMD), também conhecido como headset, e dipositivos de
interação. Google Cardboard222https://arvr.google.com/cardboard/, Oculus Quest
2333https://www.oculus.com/quest-2/ e HTC Vive Pro
2444https://www.vive.com/us/product/vive-pro2/overview/ são exemplos de VR
headsets enquanto Microsoft HoloLens 2555https://www.microsoft.com/en-
us/hololens é um exemplo de MR headset. Geralmente, XR headsets além de
fornecerem feedbacks por meio da visualização de imagens em 360º e áudio
estéreo, possuem controles para as mãos a fim de possibilitar a interação com
o ambiente virtual. Sensores que capaturam o movimento das mãos (Leap
Motion666https://www.ultraleap.com/product/leap-motion-controller/) e do corpo
(Omni One777https://invest.virtuix.com/) podem compor ao ambiente virtual e
melhorar a experiência imersiva. Dispositivos que permitem experimentar o
cheiro e gosto, bem como sentir o peso dos objetos virtuais ainda são
protótipos acadêmicos (see Section VI-C1). Os ambientes virtuais, ferramentas
de autoria e outros, também fazem parte dos XR systems.

####  II-3 Virtual Environment (VE)

Também conhecido como mundo virtual – é um espaço tridimensional gerado por
computador, onde usuários interagem entre si (por meio de avatares) ou com
outros objetos virtuais [biocca2013communication]. sherman2003understanding
[sherman2003understanding] definem como o conteúdo de algum meio. Os objetos
virtuais podem manifestar certos atributos, tais como geometria, cores,
texturas, iluminação, características dinâmicas, restrições físicas e
atributos acústicos [fuchs2011virtual].

Em alguns VEs podem ser utilizados precisões geométricas, tais como cores,
texturas e iluminação, as quais são elementos importantes para simular o mundo
real. No entanto, certos VEs não referenciam ao mundo real, constituindo-se
apenas de modelos abstratos. Independentemente do tipo de ambiente, as cores,
texturas e iluminação são elementos importantes para uma boa visualização e
imersão. Há situações em que o VE é utilizado para avaliar apenas alguma
simulação comportamental, na qual a precisão do comportamento é mais
importante que a fidelidade visual. É o caso de reações químicas, que podem
usar representações simples das moléculas baseadas em esferas coloridas, por
exemplo [fuchs2011virtual].

####  II-4 Immersive Learning (iL)

A definição de iL é uma das questões de pesquisa deste estudo secundário.
Pesquisadores em iL possuem entendimentos divergentes sobre conceitos
triviais, tais como imersão e presença, e isso nos motivou obter evidências da
literatura com relação ao entendimento de iL. Considerando nossa experiência,
entendemos iL como um método educacional que utiliza tecnologia imersiva como
meio a fim de apoiar a melhoria dos resultados de aprendizagem. Esta definição
é similar ao mobile learning (m-learning). A diferença está no aparato
tecnológico aplicado ao ensino.

##  III Related Work

Com o intuito de sistematizar a seleção de trabalhos relacionados foi
realizada uma busca por estudos através de um protocolo de revisão
simplificado. Em Janeiro de 2022, executamos a seguinte string de busca
(adaptada do estudo terciário de kitchenham2009systematic
[kitchenham2009systematic]): ( TITLE-ABS-KEY ( “immersive learning” OR
“immersive education” ) AND TITLE-ABS-KEY ( “review of studies” OR “structured
review” OR “systematic review” OR “literature review” OR “literature analysis”
OR “in-depth survey” OR “literature survey” OR “meta-analysis” OR “past
studies” OR “subject matter expert” OR “analysis of research” OR “empirical
body of knowledge” OR “overview of existing research” OR “body of published
research” ) ) AND ( EXCLUDE ( DOCTYPE , “cr” ) ). Apenas a máquina de busca
Scopus foi utilizada, pois indexa uma variedade de bibliotecas digitais. Além
disso, não é o nosso foco realizar rigorosamente a seleção de outros estudos
secundários relacionados a este. Como resultado da busca, foram retornados 16
documentos, sendo 7 estudos secundários em iL, os quais serão descritos a
seguir.

wu2020effectiveness [wu2020effectiveness] and snelson2020educational
[snelson2020educational] investigaram a performance de aprendizagem por meio
das tecnologias imersivas. Mais especificamente, wu2020effectiveness
[wu2020effectiveness] compararam os efeitos da immersive VR com non-immersive
VR. Como resultado, os 35 estudos analisados indicaram que immersive VR é mais
eficaz do que non-immersive VR. Além disso, também identificaram que immersive
VR tem grande impacto em K-12 learners; STEM (Science, Technology, Engineering
and Mathematics) e no desenvolvimento de habilidades específicas e na
simulação de situações reais. Por outro lado, snelson2020educational
[snelson2020educational] possuem o enfoque em aplicações que utilizam
equipamentos de baixo custo por meio de vídeos em 360º. Os autores
investigaram como os vídeos em 360º são usados e quais são as suas vantagens e
desvantagens na educação.

huang2021systematic [huang2021systematic] realizaram uma revisão sistemática
com objetivo de encontrar estudos primários que relatam o uso de AVR para o
ensino de idiomas. Foram encontrados 88 artigos publicados em 2011 e 2020, os
quais foram analisados nas seguintes perspectivas: ferramentas utilizadas,
perfil dos estudantes, principais achados, motivo pelo qual AVR são utilizados
e suas implicações. O estudo conclui, principalmente, que AVR eleva o nível de
aprendizagem; universitários são os principais usuários das tecnologias
imersivas e os benefícios encontrados são a melhoria dos resultados de
aprendizagem e aumento da motivação.

ntaba2021open [ntaba2021open] focam em como as tecnologias imersivas podem
apoiar em distance learning. Mais precisamente, investigaram quais são os
desafios e como AVR é utilizada para apoiar a distance learning.

qiao2021integration [qiao2021integration] and rey2021effectiveness
[rey2021effectiveness] têm o enfoque em treinamento. qiao2021integration
[qiao2021integration] investigaram a eficácia de immersive virtual reality
simulation in interprofessional education. Dentre os 12 estudos primários
selecionados, concluiu-se que tecnologias imersivas valorizam a abordagem de
aprendizado compartilhado e em equipe. rey2021effectiveness
[rey2021effectiveness] sintetizaram os critérios de resultados para medir a
eficácia do treinamento work at heights com VR em vários contextos. Dos 21
documentos analisados, os resultados apoiam safety managers and practitioners,
fornecendo um catálogo de métodos de treinamento, efeitos e indicadores de
avaliação.

Por fim, morgado2020unifying [morgado2020unifying] realizaram uma revisão de
estudos secundários e produziram um protocolo de revisão da literatura
especificamente para o escopo de iL.

De modo geral, os trabalhos acima buscaram evidências da melhoria dos
resultados de aprendizagem após a intervenção com as tecnologias imersivas.
Cada estudo focou em um contexto, domínio de aplicação e tipo de tecnologia
imersiva. Nosso estudo difere dos demais, pois estamos interessados em obter o
estado da arte de frameworks que corroboram para o avanço das pesquisas em iL,
sendo modelos de causa e efeito de variáveis que influenciam a aprendizagem,
bem como diretrizes para apoiar o desenvolvimento de ambientes educacionais
imersivos e boas práticas de uso pelos educadores e alunos. Portanto,
consideramos a ausência de uma revisão sistemática sobre iL frameworks como
uma lacuna da literatura que deve ser preenchida.

##  IV Research Method

The research method of this secondary study follows three main phases of a
systematic literature review proposed by kitchenham2007guidelines. The first
phase is associated with planning the review, in which the protocol is
developed and evaluated. Once the protocol is defined and validated by the
researchers involved, it begins the phase of conducting the review, in which
the objective is to select the primary studies, extract and perform the data
synthesis. Finally, the last phase defines the mechanisms for the
dissemination of the results found with the study. The review process is
detailed below.

###  IV-A Research Questions

In this study, the following main research question was defined: what is the
state of the art of immersive learning frameworks? A framework is understood
as a supporting structure that aim to guide the achievement of immersive
learning objectives. In order to answer this main question, secondary research
questions were defined:

  * •

RQ1: What definitions of immersive learning were adopted in primary studies?

  * •

RQ2: What are the purposes of use by the frameworks?

  * •

RQ3: What are the elements that compose the frameworks?

  * •

RQ4: What are the methods used to validate the frameworks?

The purpose of RQ1 is to identify the meaning of immersive learning used by
the authors, since its definition is not consolidated by the technical
literature. RQ2 aims to understand how frameworks support the use of Extended
Reality in teaching, for example, frameworks support the development of
immersive applications or the use of virtual environments, such as Second
Life. One of the main contributions of this review is related to RQ3.
Immersion, sense of presence and flow, among others, are common terms in this
area, but they have ambiguous definitions. For example, Slater and Mel
[slater2003note] state that immersion is related to the characteristics of
immersive devices, while jennett2008measuring define that it is associated
with cognitive issues. In this way, this research question aims to identify
the main elements that compose each framework, as well as the meaning of the
concepts and theoretical background that contributed to the design of the
frameworks. Finally, RQ4 has assessed the purpose of understanding how the
frameworks were assessed.

###  IV-B Search Process

It was established as a search process the construction of a search string
that automatically returns articles in the Scopus, IEEE Xplore, ACM Digital
Library, Science Direct and Web of Science databases. In order to support the
definition of the search string, a set of terms was established following the
PIO paradigm [kitchenham2007guidelines]:

  * •

Population: immersive education, immersive learning, immersive teaching,
immersive training;

  * •

Intervention: augmented reality, mixed reality, virtual reality, extended
reality;

  * •

Outcome: framework, design, guideline, model.

The “OR” boolean operator was used to join the related terms and the “AND”
boolean operator to join the terms of population, intervention and outcome. In
addition, “NOT” boolean operator was used as a filter strategy to avoid
articles on artificial intelligence without the context of human learning
[radianti2020systematic]. In this way, the search string is defined as:
(“immersive education” OR “immersive learning” OR “immersive teaching” OR
“immersive training”) AND (“augmented reality” OR “AR” OR “mixed reality” OR
“MR” OR “virtual reality” OR “VR” OR “extended reality” OR “XR”) AND
(framework OR design OR guideline OR model) AND NOT (“artificial intelligence”
OR “deep learning” OR “machine learning” OR “neural network”).

The search string has been validated in the Scopus database to be able to
return the following control articles: [PSdengel2020immersive,
PSgupta2019design, PSip2018design, PSklippel2020immersive,
PSschott2018virtual].

These control articles were defined by four reviewers: one professor and
researcher with large experience in experimental software engineering; two
postdoctoral researchers; and one doctoral student. All reviewers are
interested in immersive technologies in software engineering education. After
this validation, the search for the articles started.

###  IV-C Selection Criteria and Procedure

This section describes the conduction of the review phase. In November 2021,
the search string was executed in the title, abstract and keywords metadata
for each database. In the end, 1721 results were obtained: ACM (127), IEEE
Xplore (841), Science Direct (163), Scopus (277) and Web of Science (313). In
order to start the selection procedure, the following exclusion criteria were
applied by reviewers while reading title, abstract and keywords:

  * •

EC1: Duplicate article;

  * •

Language other than English;

  * •

EC2: Article not being a primary study;

  * •

EC3: Article being a work in progress or short paper;

  * •

EC4: Article not published in journal, conference or book chapter;

  * •

EC5: Authors having a most recent article;

  * •

EC6: Article not reporting as main contribution generic method or approach
that supports the development or selection of immersive educational
applications.

Fig. 2 presents the number of articles excluded according to each exclusion
criterion. After applying these criteria, 28 studies were eligible for full
text reading and the following inclusion criteria were applied:

  * •

IC1: Article being accessible for download;

  * •

IC2: Full text article written in the English language;

  * •

IC3: Article answers at least one research question from the review.

As a result, 12 articles were selected. Fig. 3 presents the number of articles
that met each inclusion criterion. During the reading of each article, three
steps were performed at the same time: data extraction; quality assessment and
snowballing. For each article, the one-level backward snowballing technique
[wohlin2014guidelines] was carried out in order to identify other studies
potentially relevant for this secondary study through bibliography references.
The first two steps were applied for each article selected by the snowballing.
At end, 3 studies were added to this review.

After the inclusion criteria and snowballing, 15 articles were selected to
compose the final set of articles for this secondary study. Fig. 4 shows all
the steps taken to find the final set of articles. The organization of the
steps was inspired by the PRISMA method [moher2009preferred].

Figure 2: Filtering process according to the exclusion criteria. Figure 3:
Filtering process according to the inclusion criteria. Figure 4: Overview of
the filtering process.

An electronic spreadsheet was used to support the data extraction process as
well as the quality assessment. The quality of selected articles was evaluated
according to the questions:

  * •

QA1: How clear was the framework’s purpose?

  * •

QA2: How well was the way of using the framework described?

  * •

QA3: How well were the framework elements described?

  * •

QA4: How well was the framework validation performed?

All researchers reviewed each article’s score, according to the following
scale (one value per question): 0 - poorly; 0.5 - reasonably; 1 - well.
Considering this score, 2 studies reached 4 points, 3 studies reached 3.5
points, 3 studies reached 3 points, 2 studies reached 2.5 points, 3 studies
reached 2 points and 2 studies reached 1 point. Regarding quality questions,
QA1 was attended by 84% of studies, QA2 by 47% of studies, QA3 by 75% of
studies, and QA4 by 63% of studies. Table I presets the list of articles and
the total score. Despite the low score, we decided to maintain the studies
because we have identified several gaps that can produce interesting
discussions and insights for future research, mainly from the perspective of
using and validating the frameworks.

TABLE I: Quality Assessment Total Score Score | Articles
---|---
1 |  [PSchan2019affordances, PSfowler2015virtual]
2 |  [PSaguayo2020framework, PSmisbhauddin2018vredu, PSdalgarno2010learning]
2.5 |  [PSschott2018virtual, PSgupta2019design]
3 |  [PSklippel2020immersive, PSkoutromanos2021mobile, PSklippel2020immersive]
3.5 |  [PSabdelaziz2014immersive, PSlee2010does, PScardona2019architectural]
4 |  [PSde2010learning, PSip2018design]

Briefly, Table II shows the number of studies selected through the sources,
studies excluded according to the inclusion and exclusion criteria and,
finally, the studies selected for data extraction.

TABLE II: Studies Selected and Included Source | #papers | #papers | #final
---|---|---|---
| selected | excluded | papers
ACM | 127 | 127 | 0
IEEE Xplore | 841 | 837 | 4
Science Direct | 163 | 163 | 0
Scopus | 277 | 270 | 7
Web of Science | 313 | 312 | 1
Snowballing | 3 | 0 | 3
Total | 1721 | 1708 | 15

###  IV-D Threats to Validity

Despite the contribution of this study, we identified some threats to
validity. The analysis is based on the 15 selected articles. For a secondary
study, this number of articles can compromise the results. Immersive learning
is a recent area of research and this fact may explain the amount of studies.
Another factor that should also be considered is the use of the term
“immersive learning” and its variations in the search string. A search with
related terms, such as virtual learning worlds, immersive learning
experiences, and others, could lead to a large volume of articles that would
be out of scope One of the main reasons for keeping the focus of our study was
to obtain works that respond at least to RQ1. Furthermore, the research method
was based on systematic literature review guidelines
[kitchenham2007guidelines] to ensure the quality of this study.

For out of 15 articles, 4 did not validate the approach and this factor can be
considered a threat to validity. These articles were kept with the aim of
obtaining the maximum amount of studies and achieving a more assertive
overview of the area. Furthermore, even though they did not meet some defined
quality criteria, they are studies published in journals and conferences and
reviewed by the scientific community peers.

##  V Results

The previous section we presented how the 15 primary studies were selected,
that is, what sources were used, search string defined, inclusion and
exclusion criteria and other details of the selection process. In this
section, we will answer each research question based on the data extracted
from the primary studies.

###  V-A What definitions of immersive learning were adopted in primary
studies (RQ1)?

Immersive technologies as support to education has been used by scientific
community for decades. There is a range of studies that adopted augmented and
virtual reality as an improving mechanism of the learning outcomes in many
areas of knowledge [bacca2014augmented] [mikropoulos2011educational].
Unfortunately, the lack of consensus on concept definition is not something
new in augmented and virtual reality. Many researchers have divergent
definitions about immersion and presence. From this scenario, we would like to
know what are the immersive learning definitions used in studies.

From 15 selected studies, 5 defined immersive learning with two points of
view. Firstly, the following authors believe that immersive learning is
related to, mainly, pedagogical and subjective aspects. In the case of
[PSdengel2020immersive], did not define it directly, but we understand that
immersive learning is the achievement of learning outcomes through educational
virtual environments. Therefore, the authors established variables (immersion,
presence and learning potential) influence the learning outcomes. In the study
[PSabdelaziz2014immersive] believes that the immersive learning concept
supports self-regulated, self-determined, self-controlled, informal and life-
long learning through a cognitive engagement network that starts with the
student and goes through the pro-action engagement, acting engagement,
reflection engagement, and reaction phases.

On the other hand, immersive learning is defined considering technological
aspects [PSklippel2020immersive, PScardona2019architectural]. Immersive
learning is immersive experiences for place-based education
[PSklippel2020immersive]. In other words, it is to support the learning
through immersive virtual field trips. According [PScardona2019architectural],
the users must achieve their learning objectives through a transfer of
immersive learning based on virtual reality to the real world with real
situations through hands-on activities, interacting with objects and events in
the simulated world.

A definition that is between the two points of view above is used by
[PSip2018design]: immersive learning is to use technologies, especially
computer graphics and human-computer interaction technologies, to create
simulated virtual worlds, in which learning can take place by employing
appropriate instructional and pedagogical approaches. The authors consider
technological and pedagogical aspects.

We believe that understanding the definition of immersive learning is very
important for the advancement of future research. Through the findings, we
realized that there is no consensus about what is immersive learning. Clearly
there is a separation between pedagogical and technological aspects. Although
[PSdengel2020immersive] consider the educational virtual environments, they
highlight that immersion, presence and learning potential are main variables
to achieve immersive learning. Moreover, [PSabdelaziz2014immersive] focused on
an approach based on the constructivist model. Only [PSip2018design]
highlighted the importance of pedagogical and technological aspects.

In addition to the definition given by the authors, we identified two papers
[PSdengel2020immersive, PSkoutromanos2021mobile] published in International
Conference of the Immersive Learning Research Network (iLRN) [iLRN]. This
conference aims to connect researchers, educators and developers in order to
discuss how XR can provide various opportunities for education. Thus, we also
consider immersive learning as a recent research area.

Therefore, in our point of view, immersive learning could be defined as a
research area that investigates how to improve the learning outcomes through
the relationship between the triad immersive technologies, psychological and
pedagogical aspects. Considering the main elements extracted from each
framework (see Section VI) these three aspects were confirmed.

###  V-B What are the purposes of use by the frameworks (RQ2)?

In this research question, we want to understand for what purposes the
immersive learning frameworks were proposed.

In general, from the point of view of the objective, the works were
categorized into theoretical and practical. Theoretical frameworks are models
that establish the relationship between factors that influence learning
outcomes or the adoption of immersive technologies, as well as elements that
support the design of learning activities in immersive educational
environments. On the other hand, we consider work that establishes guidelines
or development models that support the production of immersive educational
environments as practical frameworks.

In addition to this broad categorization between theoretical and practical
framework, we created subcategories to establish a better understanding
regarding the contribution of each work. Table III shows the classification of
theoretical frameworks and Table IV of practical frameworks.

About theoretical frameworks, the works [PSabdelaziz2014immersive,
PSde2010learning] were classified as design of learning activities, because
they define elements that must be considered to design learning activities and
assess whether immersive educational environments fit the specificities of the
activities. More specifically, [PSabdelaziz2014immersive] aims to be immersive
Web-based learning model for supporting learning through phases that virtual
worlds should provide to students to achieve learning, while
[PSde2010learning] aims to be an evaluation methodology for designing learning
activities in virtual worlds as well as evaluating learning experiences.

Factors that influence learning outcomes category classifies works that define
elements that are relate to and influence learning outcomes. In general, the
works model a causal relationship of the main elements that each author
considers important in immersive learning to explain the influence of learning
outcomes through immersive educational environments. These elements for some
authors are denominated affordances [PSschott2018virtual,
PSchan2019affordances, PSdalgarno2010learning, PSfowler2015virtual], objective
and subjective factors [PSdengel2020immersive] and variables
[PSklippel2020immersive, PSlee2010does].

In the factors that influences teachers’ intention category the model proposed
by [PSkoutromanos2021mobile] determines the teachers’ intention to use
Augmented Reality applications. This works, based on Technology Acceptance
Model (TAM) [tam1989], helps to understand what are the main characteristics
that applications must have to comply with educational purposes, from the
teacher’s point of view.

Finally, regarding practical frameworks, [PSaguayo2020framework,
PSmisbhauddin2018vredu] define a set of guidelines and design principles for
immersive environment development for educational purposes. Specifically,
[PSaguayo2020framework] consider design principles and processes that can
enhance learning outcomes within free-choice settings, such as museums and
visitor centres and [PSmisbhauddin2018vredu] developed a general framework
that transports all elements of the classroom (from instructor’s point-of-
view) to the immersive virtual environment.

Lastly, in the development model category includes the works that minimally
define a development process to be followed (steps), the actors involved
(students, instructors and developers), the types of immersive technologies,
as well as software design tools [PSgupta2019design, PSip2018design,
PScardona2019architectural].

TABLE III: Classification of Theoretical Frameworks Categories | Primary Studies
---|---
Design of learning activities |  [PSabdelaziz2014immersive, PSde2010learning]
Factors that influences learning outcomes |  [PSdengel2020immersive, PSschott2018virtual, PSchan2019affordances, PSdalgarno2010learning, PSfowler2015virtual,
|  PSlee2010does]
Factors that influences teachers’ intention |  [PSkoutromanos2021mobile]
TABLE IV: Classification of Practical Frameworks Categories | Primay Studies
---|---
Guideline |  [PSaguayo2020framework, PSmisbhauddin2018vredu]
Development model |  [PSgupta2019design, PSip2018design, PScardona2019architectural]

###  V-C What are the elements that compose the frameworks (RQ3)?

Once the frameworks have been categorized according to their objective as per
the previous section, in this research question we are interested in
understanding what are the main elements that composes each framework.

During data extraction, the elements of each framework were categorized in
order to group similar aspects among them and, mainly, to assist in data
analysis. This categorization emerged during the reading of the works,
considering the reviewers’ experience in the areas of immersive technology and
education. When it comes to immersive technology, two fundamental aspects must
be considered when developing an application: technological (devices,
infrastructure, platform etc.) and psychological (the feeling of being present
in the virtual world, emotions, awareness etc.). In education, the main aspect
considered in this work context was the pedagogical. Therefore, the frameworks
elements were grouped into technological, psychological and pedagogical
aspects, in addition to the combination between them. Appendix A shows all
tabulated data considered for this analysis.

Figure 5: Overview of the framework elements.

Fig. 5 presents an overview of the result of categorizing the frameworks
elements. Therefore, 38.10% of the works consider technological aspects,
34.52% consider pedagogical aspects and 21.43% psychological aspects. In some
cases, elements addressed more than one aspect. Thus, 3.57% represent
technological and pedagogical aspects, 1.19% represent technological and
psychological and, finally, 1.19% are related to technological, psychological
and pedagogical aspects.

Through this analysis, in most of the works, the concepts of immersion and
sense of presence were approached, but with divergent views. Some of them
consider immersion as a technological property and others as a mental state of
belonging to the virtual world. In addition, some authors claim that immersion
and a sense of presence can be considered as a psychological aspect, that is,
a feeling of belonging to the virtual world. These and other findings will be
discussed in detail in Section VI-B.

###  V-D What are the methods used to validate the frameworks (RQ4)?

Finally, the last research question aims to understand how the frameworks were
validated. Out of 15 works, 4 did not validate the proposal. Table V presents
a list of the works and the type of validation.

TABLE V: Frameworks Validation Categories | Primay Studies
---|---
Action research |  [PSklippel2020immersive, PSaguayo2020framework]
Post-test |  [PSabdelaziz2014immersive]
Pre- and post-test |  [PSdengel2020immersive, PSschott2018virtual, PScardona2019architectural, PSchan2019affordances, PSkoutromanos2021mobile, PSlee2010does]
Pre- and post-test and |  [PSip2018design, PSde2010learning]
observation of student performance |

Most studies adopted the validation strategy through pre- and post-tests.
Participants answer a questionnaire (pre-test) to record their knowledge
before the intervention, as well as obtain demographic data. Afterwards, the
participants perform some tasks and, at the end, answer another questionnaire
(post-test).

Some works chose to add observation of students performance to the pre- and
post-test. [PSde2010learning] used video observations of the real world and
the in-world sessions as well as recordings and chat logs and [PSip2018design]
lexical analysis of the learners’ comments.

The works [PSklippel2020immersive, PSaguayo2020framework] used the strategy
that can be classified as action research. Specifically,
[PSklippel2020immersive] carried out a set of evaluations and the results were
used to evolve their proposal and [PSaguayo2020framework] defined their
guidelines based on feedback during the development of immersive applications.

Finally, [PSabdelaziz2014immersive] selected a group of students and divided
them into a control group and an experimental group. Each group was selected
according to the already known profile and at the end of the tasks a post-test
questionnaire was used.

##  VI Discussion

In this section we discuss the main findings of this review and also propose
some issues to include in the research roadmap on immersive learning
frameworks.

###  VI-A Quality Assessment

###  VI-B What is immersion and presence?

One of the main characteristics of Virtual and Augmented Reality is to provide
to the user the feeling of belonging to the virtual environment. Real-time
interaction and the human senses stimulated by the device’s sensors enhance
the feeling of presence in a virtual environment. Immersion and presence are
concepts found in frameworks and their definitions are antagonistic, that is,
some authors have divergent understandings about these concepts.

For example, [PScardona2019architectural, PSdalgarno2010learning,
PSdengel2020immersive, PSklippel2020immersive, PSlee2010does] understand that
immersion corresponds to the properties and capabilities of technology to
stimulate the human sensory system. All these works were based on Slater’s
works [slater2003note, slater1997framework, slater1999measuring,
slater2004colorful]. From this point of view, immersion is a quantifiable
description of the technology, that is, one must consider the quality of the
graphic display, stereo audio, haptic sensor, motion sensor, among other
characteristics of the device.

By focusing in detail on this idea of immersion, the authors are concerned
with establishing which technological capabilities the devices provide in
order that experiences can match expectations of interaction with the virtual
environment.

Considering the Milgram’s continuum [milgram1994taxonomy], for the user to can
experience the sensation of fully belonging to the virtual environment,
devices that stimulate all human senses are needed. The most common senses
provided by XR headsets, are sight and hearing. Oculus Quest 2, HTC Vive Pro 2
and Microsoft HoloLens 2 are examples of a range of devices of this type, in
addition to allowing interaction with the virtual environment through controls
for each hand. Some devices have limitations, such as Google Cardboard, in
which the user’s interaction with the virtual environment is through a fixed
pointer on the screen, denominated gaze point. Thus, to interact with some
object in the virtual environment, the user must position the pointer with the
movement of the head and wait for the timer (a few milliseconds) until the
interaction is completed.

XR headsets have evolved over the years and have become more accessible to the
public due to reduced cost and improved technology. There are devices that
stimulate other human senses and have more intuitive interactivity, but some
of these require high investment for acquisition or are still prototypes. Omni
One is a system produced by Virtuix in which it captures human body movements
such as walking, running, jumping and rotating the body, allowing the user to
have a greater immersive experience when moving naturally in the virtual
environment. Another example of an interaction device is Leap Motion, which
allows the user to interact with the virtual environment by capturing hand
movements. Another human sense that can be explored is smell.
ION888https://ovrtechnology.com/technology/ is a device produced by OVR
Technology that allows the user to smell it during the immersive experience.
In addition to smell, prototypes are being developed to allow the user to feel
the taste, as is the case with the work of Karunanayaka et al. (2018)
[karunanayaka2018new]. Kim et al. (2020) developed a prototype to sense the
temperature of virtual objects. Finally, HaptX Gloves DK2999https://haptx.com/
is a haptic device which allows the user to feel the contact and weight of
virtual goals when interacting with them.

The works [PSschott2018virtual, PSchan2019affordances] understand that
immersion can be defined as a mental state in which the user is surrounded by
another reality demanding his/her attention.

This definition of immersion is similar to the concept of presence, also
called by some authors as a sense of presence. All works that define the
concept of presence are unanimous in stating that it is a user’s mental state
of belonging to the virtual environment in which they are interacting.

For [PSfowler2015virtual], immersion goes beyond technological and
psychological points of view. In his framework, the learning experience is
also considered, called pedagogical immersion, in which it is the pedagogical
state that arises from learning in a virtual environment.

The above works explicitly defined the understanding of immersion and presence
according to the application context. Appendix B presents the definitions of
each framework, as well as the reference used for each concept. Although some
works have not explicitly defined immersion and presence, the meaning of these
concepts can be understood throughout the reading of the full text, as in the
case of [PSip2018design, PSabdelaziz2014immersive, PSde2010learning] in which
the understanding is that immersion is a mental state, contrary to
[PSgupta2019design, PSaguayo2020framework, PSmisbhauddin2018vredu], who
understand immersion as a technological aspect. Exceptionally,
[PSkoutromanos2021mobile] do not address these concepts.

Through the discussions above, it can be seen that the concept of presence is
well defined, while immersion has several understandings. In order to
understand the relationship between the definitions of the concepts with the
objective of each framework, Table VI presents the mapping between the
framework categories, as well as the subcategories, with the types of
immersion. Most of the works that consider psychological immersion are
theoretical frameworks and the works that consider technological immersion are
equally grouped between theoretical and practical frameworks. In particular,
all theoretical frameworks that consider technological immersion are
concentrated in the influences learning outcomes category. Only
[PSfowler2015virtual] defined immersion as a set of technological,
psychological and pedagogical aspects.

TABLE VI: Immersion Mapping Category |  Subcategory |  Psychological Immersion |  Technological Immersion |  Technological psychological and pedagogical Immersion
---|---|---|---|---
Practical Framework |  Guideline |  |  [PSaguayo2020framework, PSmisbhauddin2018vredu] |
|  Model Development |  [PSip2018design] |  [PSgupta2019design, PScardona2019architectural] |
Theoretical Framework |  Design of Learning Activities |  [PSabdelaziz2014immersive, PSde2010learning] |  |
|  Influences Learning Outcomes |  [PSchan2019affordances] |  [PSdengel2020immersive, PSklippel2020immersive, PSdalgarno2010learning, PSlee2010does] |  [PSfowler2015virtual]
|  Influences Teacher’s Intention |  [PSschott2018virtual] |  |

Through the above analysis, we observe that the concept of immersion will vary
according to the purpose of each framework. For example, the models of
[PSdengel2020immersive, PSklippel2020immersive, PSdalgarno2010learning,
PSlee2010does] establish which variables influence learning outcomes in
virtual environments. In order to isolate the characteristics of the devices
with their ability to “immerse” the user in the virtual environment, the term
immersion was defined as a technological aspect and presence as a
psychological aspect of belonging to the virtual environment. From this point
of view, an immersive device is not a guarantee to provide the user with a
complete sense of presence, because it depends on other factors, such as the
proper functioning of the interaction between the user and the virtual
environment, motion sickness, fidelity in the graphical representation,
interference from the environment external, among others. The works
[PSip2018design, PSschott2018virtual, PSabdelaziz2014immersive,
PSchan2019affordances, PSde2010learning] prioritize other variables and
consider immersion as a psychological state of belonging to the virtual
environment (psychological immersion).

In our view, immersion should be considered as a technological aspect and
presence as a psychological aspect of belonging to the virtual environment. In
this way, we believe to facilitate the understanding of these trivial concepts
and support the identification of the potential of devices and virtual
environments to “immerse” the user and transmit a sense of presence.
Therefore, the greater the involvement of the human senses together with the
human-computer interaction intuitive, the greater the degree of immersion and
potentially the user will achieve the sense of presence. For example, Oculus
Quest 2 has the greatest potential to provide presence compared to Google
Cardboard, meaning the former is more immersive than the latter, however the
reach of the sense of presence depends on several factors throughout the
immersive experience.

Therefore, we conclude that immersion must be considered as an objective
aspect that characterizes the technological capacity to evoke the user’s
feeling of presence in a virtual environment and presence a subjective aspect
in which the user believes “being there” in the virtual environment that
he/she is interacting with.

###  VI-C Immersive Learning Frameworks Gaps

In this review, we identify frameworks with specific purposes, some of which
model the causal relationship between factors that can influence learning
outcomes, methods for designing activities in virtual environments, as well as
guidelines and models that define the main requirements of virtual educational
environments.

Understanding what and how factors influence learning outcomes is important to
ensure that the teaching and learning process is successful. Through
theoretical frameworks one is able to understand that there are several
factors involved in immersive learning, such as sense of presence, educational
context, learning strategies, pedagogical theories and others [CITING WORKS].
Pedagogical and psychological aspects are fundamental in immersive learning.
The practical frameworks identified in this review are limited, as their
contributions are context-specific and it is not possible to generalize. In
general, frameworks define “what” and not “how” to develop applications for
immersive learning. More specifically, the works do not detail nuances of
immersive technology in order to guide the development of virtual environments
from the point of view of  frameworks that address the details of technology
to support development, and more practical approaches that support teacher and
instructor decision-making.

As regards the frameworks analyzed in this secondary study, in the identified
gaps, as well as the experience of reviewers in Immersive Learning and
Software Engineering, we list below some aspects that are fundamental to
support the development for immersive learning, which can be considered as
research roadmap.

####  VI-C1 Level of Immersion

Since human beings have five senses (smell, taste, sight, hearing and touch)
to interact with the world they live in, researchers have sought to make users
interact with virtual environments in the same way as they interact with the
real world, making the immersive experience more complete. Thus, it is
important to define which human senses will be involved during the immersive
learning experience and which will be the forms of interaction with the
virtual environment. This decision will directly influence the choice of
immersive devices.

####  VI-C2 Immersive Devices

We consider traditional devices as multimedia (desktop, tablet and smartphone)
and mulsemedia [ghinea2014mulsemedia] as immersive devices that raise the
level of multimedia immersion and add other human senses such as smell, taste
and touch, in addition to providing more natural and intuitive interactions.
Examples of mulsemedia devices are XR headsets, haptics, motion sensors and
others. Each immersive device has characteristics that will influence the
experience as a whole. For example, the immersive experience via smartphone is
more limited compared to the XR headset.

####  VI-C3 Development Tools

Developing for XR is complex, because it needs a multidisciplinary team that
involves skills such as coding, game design, 3D modeling, storytelling, user
experience and others. For each specialty a set of tools is needed to produce
the artifacts. For example, to create 3D objects and scenarios, it is
necessary to master 3D modeling tools such as
Blender101010https://www.blender.org/, 3DS Max111111https://www.autodesk.com/
products/3ds-max/ and Maya121212https://www.autodesk.com/products/maya/.
Software engineers are not required to master these tools, nor have the
ability to model 3D objects, but they must be able to specify the trivial
development tools related to the chosen platforms. Therefore, priority must be
given to which platform the application will run on, that is, whether it will
be a native or web application.

Then the development environment to implement the virtual environment features
must be chosen. Most immersive device manufacturers provide the Software
Development Kit (SDK) according to development environments such as Android,
iOS, Web, Unity131313https://unity.com/,
Unreal141414https://www.unrealengine.com/ and others.

Therefore, if Google Carboard will be used in an immersive experience and the
virtual environment must be downloaded to the smartphone (native), the
developer must choose the development environment that they are corresponding
familiar with (Android NDK, iOS or Unity) and import the SDK. If the virtual
environment is run via browser, it is necessary to choose a tool compatible
with the parameters of the virtual environment so that the immersive
experience is not impaired. Examples of web frameworks are
WebXR151515https://immersive-web.github.io/, A-Frame161616https://aframe.io/,
Babylon.js171717https://www .babylonjs.com/ and React 360.

####  VI-C4 User Experience

Unlike applications based on the WIMP interface (Windows, Icons, Menus and
Pointers), immersive applications need attention to avoid uncomfortable
experiences. Instructions on how to interact with the virtual environment must
be accessible at all times. Oculus Quest 2 Controller, for example, has 6
buttons for each hand and this can be a lot of information for the user.
Therefore, the environment must provide a training section so that the user
can gradually get used to the virtual environment. In addition, the virtual
environment must maintain a stable frames per second (fps) rate, preferably 60
fps, to keep camera movement in the environment corresponding to the user’s
head movement, avoiding discomfort. For this, a series of restrictions in the
development is recommended, such as limiting the amount of polygons; using
just one camera instead of post-processing effects for draw calls needed in
the scene; use of panoramic images (360 degrees) and others.

####  VI-C5 Simulator Sickness

Simulator sickness is a very important aspect in XR and one that no immersive
learning framework has addressed. Users may experience uncomfortable symptoms
(such as eyestrain, fatigue, dizziness, ataxia) that make the immersive
experience difficult. Regan and Ramsey (1994) [regan1994frequency] found that
individuals exposed to the virtual environment had symptoms for up to 5 hours
after the experience. The severity and duration of these symptoms can be
influenced by the time of exposure to the virtual environment and the
intensity of the experience [kim2005characteristic]. Thus, the way the user
will move in the virtual environment is an important precaution to avoid
discomfort during and after the experience. In this example, implementing the
teleportation technique reduces the probability of the user presenting motion
sickness symptoms instead of walking freely through the virtual environment.

Below, we list some aspects to support teachers and instructors in adopting
immersive teaching experiences.

####  VI-C6 Immersive Platforms

Immersive Learning supports teaching in any field of knowledge. Therefore,
teachers and instructors do not have the skills to develop applications and
therefore need tools to support their classes. There is a range of platforms
that provide immersive content.

Talespin’s training platform181818https://www.talespin.com/ puts the user
directly into a guided scenario, in a realistic two-person discussion
situation. Engage VR191919https://engagevr.io/ and
Unimersiv202020https://unimersiv.com/ are online training and education
platforms that have immersive content from various areas of knowledge.

In addition, teachers and instructors can create virtual spaces and insert
their contents. Frame VR212121https://framevr.io/ and Mozilla
Hubs222222https://hubs.mozilla.com/ are examples of current web tools that
allow one to create virtual classrooms where students can access
simultaneously through avatars, communicate and interact with each other.
Furthermore, Second Life232323https://secondlife.com/ and Open
Wonderland242424http://openwonderland.org/ are virtual spaces that have been
much explored by the scientific community.

####  VI-C7 Available Infrastructure

Using immersive experiences in teaching still emerges as a challenge, as
immersive equipment still requires considerable investment. Therefore, the
choice of device that will be used in the teaching and learning process
directly impacts the pedagogical performance. Desktops and smartphones are
more common devices among people than XR headsets. On the other hand, the
educational institution can choose to purchase immersive devices, but it will
require a high investment.

Another point is the hardware configuration and Internet connection speed.
Virtual environments demand high performance in image processing and take up a
lot of storage space. In the case of a web application, the connection quality
will also impact the performance of learning activities.

####  VI-C8 Improved Learning Outcomes

This is one of the great challenges in immersive learning. Ensuring improved
learning outcomes through immersive experiences still requires further
empirical studies. Increasing the degree of immersion of devices is not a
guarantee of high academic performance. The sense of presence and pedagogical
aspects (quality of instructional content and pedagogical theories) are also
important and must be considered to achieve effectiveness in learning
outcomes. Therefore, further studies are needed on how to improve learning
considering technological, psychological and pedagogical aspects.

####  VI-C9 Learning Analytics

Through the analysis carried out in this study, it was identified that no
framework considered the monitoring of student learning in virtual
environments. In general, virtual environments immerse students in
instructional content, but learning data are not captured to analyze students’
educational performance. We believe that Learning Analystics can provide
valuable information about actual performance and improve the teaching-
learning process. For example, through biofeedack sensors, heart rate,
breathing, sweat, and temperature readings can indicate whether the student in
a particular section felt any discomfort [christopoulos2020learning,
kazanidis2021learning, reis2021learning].

##  VII Conclusion

This systematic literature review aimed to identify the state of the art of
immersive learning frameworks. Through the 15 selected articles it was
possible to obtain an overview of the contributions and identify gaps and
research opportunities.

Through the research questions, we identified that the authors have divergent
understandings about immersive learning (RQ1), as well as the definition of
immersion. In addition, we grouped the works regarding the purpose of use and
we found that there are frameworks to support the design of learning
activities, identify which factors influence the learning and intention of
teachers, guidelines and development models (RQ2). We also found that
frameworks are composed of three main aspects: technological, psychological
and pedagogical (RQ3). Finally, most of the frameworks were validated through
a questionnaire, but 4 articles did not validate the proposal (RQ4).

The relevance of this study lies in the discussion and definition of the
concept of immersion, better understanding of immersive learning,
identification of gaps and the proposal of a research roadmap so that
frameworks can address the development of immersive environments greater
detail, as well as the use of experiences immersive by teachers and
instructors.

As future works, the results and discussions of this systematic literature
review can be used to guide research and propose frameworks that aim to
address implementation details and, consequently, obtain more effective
virtual environments and corroborate immersive learning. fernandes2021work is
an example of a framework that supports the planning of the development of
immersive educational applications, considering the characteristics of
technologies, skills, competences and pedagogical approaches in the context of
Software Engineering Education.

## Acknowledgments

The authors would like to thank CNPq and CAPES Brazilian Agencies for
financial support.

IEEEtranN-ital primarystudies dengel2020immersive gupta2019design ip2018design
klippel2020immersive schott2018virtual abdelaziz2014immersive
aguayo2020framework cardona2019architectural chan2019affordances
dalgarno2010learning fowler2015virtual de2010learning koutromanos2021mobile
lee2010does misbhauddin2018vredu

##  Appendix A Framework Elements

Text in progress.

##  Appendix B Immersion and Presence Definitions

Text in progress.


# I Need Help! Evaluating LLM’s Ability to Ask for Users’ Support:
A Case Study on Text-to-SQL Generation

Cheng-Kuang Wu1,2, Zhi Rui Tam111footnotemark: 1, Chao-Chung Wu1, Chieh-Yen
Lin1,
Hung-yi Lee2, Yun-Nung Chen222footnotemark: 2
1Appier AI Research
2National Taiwan University
Equal contributionEqual advisorship

###### Abstract

In this study, we explore the proactive ability of LLMs to seek user support,
using text-to-SQL generation as a case study. We propose metrics to evaluate
the trade-off between performance improvements and user burden, and
investigate whether LLMs can determine when to request help and examine their
performance with varying levels of information availability. Our experiments
reveal that without external feedback, many LLMs struggle to recognize their
need for additional support. Our findings highlight the importance of external
signals and provide insights for future research on improving support-seeking
strategies.

I Need Help! Evaluating LLM’s Ability to Ask for Users’ Support:
A Case Study on Text-to-SQL Generation

Cheng-Kuang Wu1,2††thanks: Equal contribution, Zhi Rui Tam111footnotemark: 1,
Chao-Chung Wu1, Chieh-Yen Lin1, Hung-yi Lee2††thanks: Equal advisorship, Yun-
Nung Chen222footnotemark: 2 1Appier AI Research 2National Taiwan University

##  1 Introduction

The impressive instruction-following Wei et al. (2021) abilities of large
language models (LLMs) have enabled their out-of-the-box usage to solve
problems. However, these models generate hallucinated content Rawte et al.
(2023) or incorrect predictions in their efforts to fulfill user instructions,
which undermines their reliability.

When LLMs generate incorrect outputs for a given instruction, the issue can be
examined from multiple perspectives. One is that the model simply lacks the
competence to satisfy the instruction, suggesting a straightforward solution:
enhancing the model’s capabilities, which is the focus of most previous
research. Another is that the model could actually solve the task with
additional support. For instance, Pourreza and Rafiei (2023) found that models
often fail due to underspecified natural language queries. Similarly, Li et
al. (2024) showed that while GPT-4 struggles initially, its performance can
improve by up to 20.01% with human-annotated external knowledge. In such
cases, models should proactively seek help rather than attempting to satisfy
instructions with insufficient information.

Motivated by these considerations, we aim to investigate whether LLMs can
identify when to ask for user support. Since providing such support requires
additional effort from users, there is an inherent trade-off between “LLM
performance improvement from user support” and “user burden”. Therefore, we
seek to answer the following research questions: RQ1: How can we design
evaluation metrics to quantify this trade-off? RQ2: How effectively do LLMs
manage this trade-off, and what strategies are effective in improving it?

Figure 1: Overview of our experiments on text-to-SQL. LLMs struggle to
determine when they need help based solely on the instruction ($x$) or their
output ($\hat{y}$). They require external feedback, such as the execution
results ($\hat{r}$) from the database, to outperform random baselines.

In this work, we focus on the text-to-SQL task as a case study to empirically
investigate the aforementioned research questions. We chose the text-to-SQL
task for several reasons: (1) Its promising applicability, empowering lay
users to retrieve data with natural language queries. (2) The inherent
ambiguity in some natural language queries, leading to uncertainty in the
generation of SQL code Pourreza and Rafiei (2023), making it suitable for
scenarios where additional user support is beneficial. (3) There exists a
large-scale BIRD dataset Li et al. (2024) with human-annotated external
knowledge, providing a valuable source of user support for our empirical
investigation.

Our contributions can be summarized as follows:

  1. 1.

We propose metrics for evaluating the trade-off between performance
improvement from user support and the associated user burden.

  2. 2.

We conduct experiments using various methods to balance this trade-off,
providing insights into LLMs’ capabilities in seeking user support and
identifying effective strategies for enhancing their performance.

##  2 Formulation for Seeking Support

###  2.1 General Setup

Consider an LLM $f$ parameterized by $\theta$, along with a prompt template
$p(\cdot)$. Given a natural language instruction $x$, we use $z$ to represent
support, which should enhance the LLM’s ability to fulfill $x$. Formally,
$\hat{y}_{z}=f(p(x,z)\mid\theta)$ is more likely to satisfy $x$ compared to
$\hat{y}=f(p(x)\mid\theta)$. We denote the "ask for support" signal emitted by
the LLM as $\hat{a}$, defined as a confidence score in the range $[0,1]$,
where $1$ indicates an absolute need for support. A threshold $\tau$ is then
used to determine whether to request $z$. In practice, $\hat{a}$ could also be
a natural language request specifying the type of support needed by the LLM,
which we leave for future work.

###  2.2 Evaluation

To measure the trade-off between performance improvement from user support and
user burden, we need 2-dimensional evaluation. One dimension is the user
burden ($B$), which we define as the proportion of instances where the LLM ask
for support:

$B=\frac{N_{\mathrm{ask}}}{N}$

where $N_{\mathrm{ask}}$ is the number of instances where the LLM asks for
support, and $N$ denotes the total number of instances in the test set. The
other dimension is the performance improvement ($\Delta$, Delta):

$\Delta=\frac{1}{N}\sum_{i=1}^{N_{\mathrm{ask}}}(h(y_{i},\hat{y}_{i,z})-h(y_{i},\hat{y}_{i}))$

where $h(\cdot)$ is the evaluation function of a given task, which takes
ground truth $y_{i}$ and model output $\hat{y}_{i}$ as arguments
($\hat{y}_{i,z}$ is an output with the help of $z$). Inspired by the idea
behind the ROC curve Majnik and Bosnić (2013), we illustrate this trade-off
with a graph, where the performance curve is plotted by adjusting the
threshold $\tau$ from high to low along the x-axis. We refer to this curve as
Delta-Burden Curve (DBC) (see the leftmost subplot of Figure 2).

###  2.3 Methods for Seeking Support

We design a prompt template $p_{\mathrm{ask}}(\cdot)$ to enable LLMs to
request support by $\hat{a}=s(f(p_{\mathrm{ask}}(w)\mid\theta))$. Here, $w$
represents the textual information that the LLM $f$ uses to determine whether
it needs to seek support, and $s$ is the scoring function that converts the
probability distribution of output tokens into a confidence score
$\hat{a}\in[0,1]$. We propose methods with varying compositions of $w$ to
explore the information LLMs require to achieve better trade-off under DBC.
Note that $p_{\mathrm{ask}}$ remains the same across all methods to minimize
prompt engineering. An overview of these methods is shown in Figure 1.

Direct Ask (DA): $w=(db,x)$, composed of database schema $db$ and user data
requirement $x$.

Write then Ask (WA): $w=(db,x,\hat{y})$, where the LLM generates the SQL code
$\hat{y}=f(p(db,x)\mid\theta)$ first and then use this self-generated output
as the additional information in $w$.

Execute then Ask (EA): $w=(db,x,\hat{y},\hat{r})$, where the execution results
$\hat{r}$ is returned by the database by executing LLM-generated SQL
$\hat{y}$.

##  3 Experiments

###  3.1 Dataset

We use BIRD Li et al. (2024), which includes human-annotated external
knowledge that serves as $z$. For example, $z$ might be domain-specific
knowledge, such as how to calculate financial indicators from database values.
The instruction $x$ represents the users’ data requirements, paired with the
ground truth SQL $y$. It uses Execution Accuracy (EX) as the evaluation
metric, where $h(y_{i},\hat{y}_{i})$ is defined as
$\mathbbm{1}(r_{i}=\hat{r}_{i})$. Here, $r_{i}$ is the SQL execution result of
$y_{i}$, and $\hat{r}_{i}$ is the execution result of $\hat{y}_{i}$. Simply
put, EX is the proportion of testing instances where $r_{i}$ and $\hat{r}_{i}$
are identical.

Methods/LLMs | Wizard | Llama3 | DPSeek | GPT-3.5 | Mixtral | GPT-4t | GPT-4o
---|---|---|---|---|---|---|---
Random Baseline | 0.5000 | 0.5000 | 0.5000 | 0.5000 | 0.5000 | 0.5000 | 0.5000
Direct Ask | 0.4915 | 0.4834 | 0.4976 | 0.4390 | 0.5301 | 0.5758 | 0.5479
Write then Ask | 0.4759 | 0.4497 | 0.4857 | 0.4735 | 0.5677 | 0.5807 | 0.5740
Execute then Ask | 0.5096 | 0.4987 | 0.5848 | 0.6313 | 0.6242 | 0.6641 | 0.5989
Table 1: Area Under Delta-Burden Curve (AUDBC) across different methods and LLMs. Text in bold denotes the method with the best performance, while underlined text means better than random (uniform sampling of $\hat{a}\in[0,1]$). Support/LLMs | Wizard | Llama3 | DPSeek | GPT-3.5 | Mixtral | GPT-4t | GPT-4o
---|---|---|---|---|---|---|---
w/o user support | 0.1721 | 0.1767 | 0.2360 | 0.3064 | 0.2419 | 0.3142 | 0.3096
w/ full user support | 0.2764 | 0.3475 | 0.4185 | 0.4668 | 0.4126 | 0.4889 | 0.5117
Table 2: Execution accuracy (EX) of different support levels. Full user
support means $B=1$ (see Section 2.2).

###  3.2 Implementation

For open-weight LLMs, we use WizardCoder-34B Luo et al. (2023),
Llama-3-70b-chat, DeepSeek-Coder-33B Guo et al. (2024), and Mixtral-8x22B
Jiang et al. (2024) for diversity of different LLM families. For closed-source
LLMs, we use gpt-3.5-turbo-0125, gpt-4-turbo-2024-04-09, and gpt-4o-2024-05-13
OpenAI (2023). The prompt $p_{\mathrm{ask}}(w)$ (included in Appendix A)
instructs the model to output a single token Yes/No to indicate whether it
needs support. We define the scoring function $s$ as the softmax of Yes over
log probabilities of Yes and No to derive $\hat{a}\in[0,1]$.

##  4 Main Results

Using the formulation in Section 2.2, we quantify the performance of different
methods with the Area Under Delta-Burden Curve (AUDBC) in Table 1. Visualized
DBCs are available in the leftmost subplots in Figure 2. Note that AUDBC
should only be compared between methods under the same LLM, as it is
normalized to the range of $[0,1]$ by dividing the area under the curve by the
maximum square area, which depends on the scale of $\Delta$EX and differs
across LLMs, as shown in Table 2.

There are three major findings: (1) Execution then Ask consistently improves
the performance-burden trade-off for LLMs, although Llama-3-70b-chat fails to
outperform the random baseline. (2) The leftmost four LLMs in Table 1 do not
surpass the random baseline without the assistance of $\hat{r}$, indicating
that many current LLMs still struggle to determine the need for support based
on $x$ and $\hat{y}$ alone. (3) Despite this, the rightmost three LLMs
outperform the random baseline with the Write then Ask $(x,\hat{y})$ or even
Direct Ask $(x)$ methods. Nevertheless, the inclusion of $\hat{r}$ remains
beneficial for further enhancing the trade-off between performance improvement
and user burden. Practical implications of the third point include the
potential for cost savings by trading off the execution of $\hat{y}$ to obtain
$\hat{r}$ in certain resource-constrained scenarios.

##  5 Discussion

Figure 2: Performance curves of gpt-3.5-turbo-0125. Curves of other LLMs are shown in Appendix B. Methods/LLMs | Wizard | Llama3 | DPSeek | GPT-3.5 | Mixtral | GPT-4t | GPT-4o | Gemini | Claude
---|---|---|---|---|---|---|---|---|---
Random Baseline | 0.5000 | 0.5000 | 0.5000 | 0.5000 | 0.5000 | 0.5000 | 0.5000 | 0.5000 | 0.5000
EA (real logprobs) | 0.5096 | 0.4987 | 0.5848 | 0.6313 | 0.6242 | 0.6641 | 0.5989 | - | -
EA (verbalized) | 0.5011 | 0.5333 | 0.4964 | 0.5945 | 0.6226 | 0.4850 | 0.5152 | 0.5624 | 0.6174
Table 3: Area Under Delta-Burden Curve (AUDBC) with the verbalized token log
probabilities approach. Text in bold denotes the method with the best
performance, while underlined text means better than random.

###  5.1 Analysis on the Delta-Burden Curves

The Delta-Burden Curves (DBCs) plotted in Figure 2 quantify the following
practical question: Under the same user burden, which method can achieve more
performance boost? To further analyze how this performance boost is achieved,
we decompose the concept into two abilities:

  1. 1.

The ability to ask for support when the LLM cannot satisfy the instruction
originally.

  2. 2.

The ability to utilize support effectively to flip the incorrect output to the
correct output.

1\. For the first ability, we introduce the following metrics inspired by the
precision-recall trade-off:

Precision of Asking for Support ($P_{\mathrm{ask}}$) When the LLM asks for
support, it should be the case that the LLM cannot satisfy the instruction
originally, or it would cause unnecessary user burden:

$P_{\mathrm{ask}}=\frac{\\#(\text{AskforSupport}\And\text{OriginallyWrong})}{\\#\text{AskforSupport}}$

Recall of Asking for Support ($R_{\mathrm{ask}}$) When the LLM is not able to
satisfy the instruction originally, it should identify this need and ask for
support:

$R_{\mathrm{ask}}=\frac{\\#(\text{AskforSupport}\And\text{OriginallyWrong})}{\\#\text{OriginallyWrong}}$

PR Curve of Asking for Support Similar to how DBC is plotted, one can also
adjust the threshold $\tau\in[0,1]$ from high to low along the x-axis to plot
the Precision-Recall Curve of Asking for Support.

2\. For the second ability, we introduce Flip Rate:

Flip Rate: This metric is calculated as the proportion of instances where the
LLM’s initially incorrect answers were corrected after receiving support,
divided by the total number of instances where support was requested.
Formally, it is defined as:

$FR=\frac{1}{N_{ask}}\sum_{i=1}^{N_{\mathrm{ask}}}(h(y_{i},\hat{y}_{i,z})-h(y_{i},\hat{y}_{i}))$

Different from $\Delta$ defined in Section 2.2, this metric emphasizes the
efficiency of leveraging support instead of the total improvement on the test
set. Like DBC, one may adjust the threshold $\tau$ to plot the Flip Rate Curve
(FRC). With the definition of these two abilities, we plot the DBC, PR Curve,
and FRC on Figure 2. Although the Write then Ask method shows near-random
performance in DBC, the PR Curve indicates it achieves better-than-random
performance in identifying when support is needed. However, its lower Flip
Rate suggests it is less efficient in utilizing the support to correct
mistakes. These two abilities, represented by the PR Curve and FRC,
respectively, balance each other out, resulting in near-random performance on
the DBC. This finding shows that the ability to identify the need for support
and the ability to utilize that support are distinct. In future work, it is
worth exploring how to further enhance each of these abilities.

###  5.2 LLMs without Access to Log Probabilities

Given that not all LLMs provide access to token log probabilities, we discuss
how our method can be adapted for these “black-box” models. We modify the
prompt template $p_{\mathrm{ask}}$ to $p_{\mathrm{verb}}$, which instructs the
LLM to output the verbalized confidence score $\hat{a}$ directly by specifying
the range and meaning of $\hat{a}\in[0,1]$ in $p_{\mathrm{verb}}$ (attached in
Appendix A.2). In addition to the seven LLMs mentioned in Section 3.2, we also
include two black-box models: gemini-1.0-pro-001 and claude-3-haiku-20240307.
The results, shown in Table 3, indicate that using verbalized confidence
scores generally degrades performance for most LLMs. However, it remains a
promising alternative for black-box LLMs such as Gemini and Claude to surpass
the random baseline.

##  6 Related Work

The ability of LLMs to identify the need for support relies on their well-
calibratedness Kadavath et al. (2022), which refers to their capacity to
recognize uncertainty. Previous studies focus on enhancing the calibration of
predictions Xiao et al. (2022); Kuhn et al. (2023), or using verbalized token
probabilities to achieve better calibration Tian et al. (2023). Our work
extends this line of research by exploring how LLMs can effectively seek user
support by leveraging their well-calibrated property. The major distinction
between this and existing calibration studies lies in extending the focus from
identifying the uncertainty to utilizing support.

##  7 Conclusion

We propose a framework for LLMs to seek support, and evaluate methods on Text-
to-SQL generation. Our findings suggest the importance of external signals,
such as SQL execution results, in helping LLMs better manage performance-
burden trade-off. We further decompose DBC into the ability of identify the
need for support and the ability to utilize the support. Future works may
explore a broader range of tasks or develop methods to improve both the
identification and utilization of support.

##  8 Limitations

###  8.1 Task Coverage

The scope of our experiments is limited to the Text-to-SQL task. While this
task provides a useful case study for evaluating LLMs’ ability to seek and
utilize support, it does not encompass the full range of potential
applications for LLMs. Future work should extend the evaluation to a broader
set of tasks to ensure the generalizability of our findings.

###  8.2 Types of Support

In this study, we primarily focus on a single type of support: human-annotated
external knowledge. However, there are many other types of support that LLMs
might require. Future works could explore how LLMs can request and utilize
these various forms of support to enhance their performance.

###  8.3 Dependence on External Feedback

Our findings indicate that LLMs significantly benefit from external signals,
such as SQL execution results. However, this reliance on external feedback may
not always be feasible in practical applications, where immediate execution or
access to external data might be limited. Developing methods that enable LLMs
to better manage without such feedback remains an important area for future
exploration.

## References

  * Guo et al. (2024) Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024.  Deepseek-coder: When the large language model meets programming - the rise of code intelligence.  _ArXiv_ , abs/2401.14196.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L’elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024.  Mixtral of experts.  _ArXiv_ , abs/2401.04088.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022.  Language models (mostly) know what they know.  _arXiv preprint arXiv:2207.05221_.
  * Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023.  Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.  _arXiv preprint arXiv:2302.09664_.
  * Li et al. (2024) Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. 2024.  Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls.  _Advances in Neural Information Processing Systems_ , 36.
  * Luo et al. (2023) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023.  Wizardcoder: Empowering code large language models with evol-instruct.  _ArXiv_ , abs/2306.08568.
  * Majnik and Bosnić (2013) Matjaž Majnik and Zoran Bosnić. 2013.  Roc analysis of classifiers in machine learning: A survey.  _Intelligent data analysis_ , 17(3):531–558.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Pourreza and Rafiei (2023) Mohammadreza Pourreza and Davood Rafiei. 2023.  Evaluating cross-domain text-to-sql models and benchmarks.  _arXiv preprint arXiv:2310.18538_.
  * Rawte et al. (2023) Vipula Rawte, Amit Sheth, and Amitava Das. 2023.  A survey of hallucination in large foundation models.  _arXiv preprint arXiv:2309.05922_.
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023.  Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5433–5442.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Xiao et al. (2022) Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022.  Uncertainty quantification with pre-trained language models: A large-scale empirical analysis.  _arXiv preprint arXiv:2210.04714_.

##  Appendix A Prompt Templates

We include the prompt templates used in this work.

###  A.1 Prompt for Seeking Support

The prompt template $p_{\mathrm{ask}}(w)$ used to instruct LLMs for seeking
support is as follows:

You are currently doing the text-to-SQL task. Based on the information
provided ({items}), you have to determine whether additional hints are
required for you to generate the SQL correctly to answer the user’s question.
You should only ask for additional hints when you actually need them, since
you will also be evaluated based on the number of times you ask for hints,
which would be provided by the user.

information provided (enclosed by triple backticks):
“‘
{information}
“‘

Answer a single word Yes if you need hints (since the information provided is
not enough to generate SQL correctly). Answer a single word No if hints are
not required (since you are already confident to generate SQL).
Do you need additional hints? Answer (Yes / No):

In this template, the actual contents of {items} and {information} depend on
the method used. The contents are summarized in Table 4. For example,
$w=(db,x,\hat{y},\hat{r})$ in Execute then Ask (EA), so {items} will be filled
with the four item names and {information} will be replaced by actual
information of the four items. Similarly for Write then Ask
($w=(db,x,\hat{y})$) and Direct Ask ($w=(db,x)$).

Item | Item Name | Information
---|---|---
$db$ | Database schema | {db_schema}
$x$ | User’s question | {question}
$\hat{y}$ | Generated SQL | {gen_sql}
$\hat{r}$ | SQL execution results | {exe_results}
Table 4: Contents in the prompt $p_{\mathrm{ask}}(w)$, where {items} will be
filled with words in the “Item Name” column, while {information} is replaced
with actual information of text in {red}.

###  A.2 Prompt for Seeking Support (Verbalized)

The prompt template for generating verbalized probabilities in LLMs without
access to token log probabilities (e.g., Gemini and Claude families):

You are currently doing the text-to-SQL task. Based on the information
provided ({items}), you have to determine whether additional hints are
required for you to generate the SQL correctly to answer the user’s question.
You should only ask for additional hints when you actually need them, since
you will also be evaluated based on the number of times you ask for hints,
which would be provided by the user.

information provided (enclosed by triple backticks):
“‘
{information}
“‘

Do you need additional hints? Provide the precise probability that you need
hints (closer to 0 means you don’t need hints, closer to 1 means you need
hints).
Give ONLY the precise probability to five decimal places (format: 0.abcde,
where abcde can be different digits), no other words or explanations are
needed.

The prompt template is similar to the original template shown in A.1, except
that the last few sentences are modified.

###  A.3 Prompt for Generating SQL Code

The prompt template $p(\cdot)$ for converting user data requirement $x$ into
SQL code is as follows:

{db_schema}

– Using valid SQLite, answer the following questions for the tables provided
above.
– Question: {question}
Now, generate the correct SQL code directly in the format of
“‘sql\n<your_SQL_code>\n“‘:

If user support $z$ is provided (i.e., when LLMs ask for support), the prompt
template is slightly modified as follows:

{db_schema}

– External Knowledge: {support}
– Using valid SQLite, answer the following questions for the tables provided
above. You can use the provided External Knowledge to help you generate valid
and correct SQLite.
– Question: {question}
Now, generate the correct SQL code directly in the format of
“‘sql\n<your_SQL_code>\n“‘:

In these two templates, {db_schema} is $db$, {question} is user data
requirement $x$, and {support} is user support $z$, which is human-annotated
external knowledge in BIRD Li et al. (2024).

##  Appendix B Performance Curves

We present visualizations of all performance curves in Table 3, 4, 5, 6, 7, 8,
and 9.

Figure 3: Performance curves of WizardCoder-Python-34B-V1.0. Figure 4:
Performance curves of Llama-3-70b-chat-hf. Figure 5: Performance curves of
deepseek-coder-33b-instruct. Figure 6: Performance curves of
gpt-3.5-turbo-0125. Figure 7: Performance curves of
Mixtral-8x22B-Instruct-v0.1. Figure 8: Performance curves of
gpt-4-turbo-2024-04-09. Figure 9: Performance curves of gpt-4o-2024-05-13.