# Enhancing Psychometric Analysis
with Interactive ShinyItemAnalysis Modules

Patrícia Martinková1,2, Jan Netík1,2, Adéla Hladká1
1 Institute of Computer Science of the Czech Academy of Sciences, Prague,
Czech Republic
2 Faculty of Education, Charles University, Prague, Czech Republic

###### Abstract

ShinyItemAnalysis (SIA) is an R package and shiny application for an
interactive presentation of psychometric methods and analysis of multi-item
measurements in psychology, education, and social sciences in general. In this
article, we present a new feature introduced in the recent version of the
package, called "SIA modules", which allows researchers and practitioners to
offer new analytical methods for broader use via add-on extensions. We
describe how to build the add-on modules with the support of the new SIAtools
package and demonstrate the concepts using sample modules from the newly
introduced SIAmodules package. SIA modules are designed to integrate with and
build upon the SIA interactive application, enabling them to leverage the
existing infrastructure for tasks such as data uploading and processing. They
can access a range of outputs from various analyses, including item response
theory models, exploratory factor analysis, or differential item functioning
models. Because SIA modules come in R packages (or extend the existing ones),
they may come bundled with their datasets, use object-oriented systems, or
even compiled code. We discuss the possibility of broader use of the concept
of SIA modules in other areas.

##  1 Introduction

Measurement in social sciences diverges from the straightforward
quantification of physical attributes such as height or weight because the
measured traits are _latent_ , existing beyond direct observation. These
measurements consist of a considerable amount of error, which needs to be
accounted for, and multiple raters and/or multi-item instruments are involved.
Consequently, a spectrum of statistical and psychometric models and techniques
have been developed to analyze such measurements in the social sciences [98,
60, 85]. These methodologies include providing proofs of the measurement
reliability, assessing validity, and analyzing the internal structure with
factor analysis. Given that social science measurement typically entails
multiple components such as items (or criteria, raters, occasions, etc.),
there is a particular focus on modeling item responses within multi-item
measurements and checking the functioning of each individual item.

The ShinyItemAnalysis (SIA) R package [87] was developed to provide helpful
functions and an interactive platform containing a comprehensive collection of
psychometric tools designed for analyzing multi-item measurements. The
interactive SIA application (Figure 1) serves as an entry point for newcomers
to R, offering access to various sample datasets and the ability to upload and
analyze their data. The psychometric analyses within the application are
structured in sections, aligning with the workflow outlined in the _Standards_
(AERA, APA, and NCME aera2014standards) and more closely described by [85].
Furthermore, it includes sample R code employing the functions of
ShinyItemAnalysis and other standard psychometric packages such as mirt [65],
psych [99], or difR [80], to foster moving from a graphical user interface
(GUI) to the command-line interface (CLI) of R, enable automation, and
reproducibility. The SIA application also offers the automatic generation of
PDF and HTML reports, which may help incorporate psychometric analyses into
the test development process [87]. The interactive environment of SIA has the
potential to aid in teaching psychometric methods and concepts and support the
implementation of psychometric methods in R while also expanding their reach
to broader audiences.

Figure 1: ShinyItemAnalysis application intropage

As the field and data complexity undergo a rapid evolution, we aim to enable
users to extend the analyses and functionalities of the SIA application. To
achieve this, we are introducing the "SIA modules" feature. This capability
empowers researchers and practitioners to develop add-on SIA modules that
seamlessly integrate with and expand upon the capabilities of the main
application. In doing so, we take inspiration from jamovi [106], JASP [79],
and R packages Rcmdr [73], Deducer [72], and RKWard [100], all of which offer
extension frameworks, thus being similar to our endeavor. Apart from the
aforementioned software, both the SIA application and SIA modules are written
entirely in R, keeping it open for the wider R community.

The paper is structured as follows: Section 2 presents the background
information on the main statistical models utilized for the psychometric
analysis of multi-item measurements, which are already integrated within the
main SIA application, and on more complex models and methods that motivate
extensions via SIA modules. In Section 3, we describe the architecture of SIA
modules, how they interact with the core ShinyItemAnalysis package, and we
detail the process of developing new SIA modules with the aid of the SIAtools
package [94]. At the end of the section, we also present a comprehensive guide
to module development, including step-by-step instructions. Section 4 offers
an overview of existing modules housed in the SIAmodules package [89], and
examples of modules residing in other packages. Finally, Section 5 provides a
discussion concerning the implications of SIA modules and the possibility of a
broader application of the concept across other research areas.

##  2 Models and software

The main SIA application includes core psychometric models and methods for
evaluating multi-item measurement in social sciences, such as providing proofs
of measurement validity, models to assess reliability, and proper functioning
of every single item. Moreover, users can integrate additional analyses and
advanced approaches, such as computerized adaptive testing, text analysis, and
others, through the SIA modules.

###  2.1 Validity and reliability of measurement scores

The concept of validity describes the degree to which an assessment instrument
measures its intended construct. Quantitative evidence from various sources
can be employed, with various statistical models available to demonstrate and
assess measurement validity from different perspectives. One such source of
evidence is provided by an external variable (criterion) measuring the same
construct and employing correlation coefficients, $t$ tests, analysis of
variance, or regression models, depending on the data structure. Additionally,
the internal structure of the test, which is an important aspect of validity,
can be examined through factor analysis.

Reliability, on the other hand, refers to the consistency of measurements and
the extent to which they are affected by error. In the realm of multi-item
measurements, one way of assessing reliability is through the analysis of
internal consistency, which involves correlations between items or between
subscores obtained from splits of the test. Another approach entails analyzing
correlations between scores derived from multiple test administrations or
assessments by multiple raters, where applicable. Assessing reliability in
more complex data structures may necessitate variance component models,
offering a flexible alternative [86].

###  2.2 Modelling item responses in multi-item measurements

Analyzing item responses is important for developing multi-item measurements
and for a deeper understanding of the respondent’s traits and components of
the measured construct.

####  2.2.1 From traditional item analysis to IRT models

Traditional item analysis (Figure 2a.) uses proportions, percentages, and
correlations to describe item functioning and characteristics of individual
items, such as their difficulty or discrimination. This empirical analysis may
also provide information on response options, including the correct option,
the incorrect options (distractors), or points of a Likert scale. The
implementation is available within the ItemAnalysis(), DistractorAnalysis(),
and plotDistractorAnalysis() functions of the ShinyItemAnalysis package that
are accessible in the R console. While the calculations involved are
straightforward, the inference on traditional item characteristics is limited.

a. Traditional item analysis

b. Regression model

c. IRT model

Figure 2: Different approaches to item analysis on the same item

Furthermore, regression models (Figure 2b.) allow interlacing empirical
values, i.e., category selection proportions, with respect to the observed
ability, sometimes referred to as the _matching criterion_ , by a smooth
curve. Item characteristics can then be estimated as parameters of these
smooth curves, often also called Item Characteristic Curves (ICCs), for each
item separately or jointly for all items.

Finally, item response theory (IRT) models (Figure 2c.) simultaneously
estimate latent traits along with all item parameters, resulting in a more
precise description of item functioning. Additionally, they provide confidence
intervals for latent trait estimates. However, this heightened model
complexity goes hand in hand with increased computational demands,
necessitating a sufficient number of test-takers for accurate fitting.

For instance, consider modeling the probability of a correct answer or item
$i$ endorsement on a person’s $p$ ability $\theta_{p}$ using a 3-parameter
logistic (3PL) model:

$\displaystyle\pi_{pi}=\mathrm{P}(Y_{pi}=1|\theta_{p})=c_{i}+(1-c_{i})\frac{1}{1+\exp\left(-\beta_{0i}-\beta_{1i}\theta_{p}\right)}.$

In the IRT framework, $\theta_{p}$ is treated as latent and requires
estimation, whereas, in the regression model setting, it represents an
observed variable. In this parametrization, $\beta_{0i}$ describes the
intercept of the underlying ICC for item $i$, and $\beta_{1i}$ defines ICC’s
slope at the inflection point, while $c_{i}$ denotes the lower asymptote,
which can be interpreted as the probability of (pseudo)-guessing.
Alternatively, a more typical parametrization in item response modeling can be
employed:

$\displaystyle\pi_{pi}=c_{i}+(1-c_{i})\frac{1}{1+\exp\left(-a_{i}(\theta_{p}-b_{i})\right)}.$
(1)

Here $b_{i}$ represents the item $i$ difficulty, which is also the location of
the inflection point of the modeled ICC, $a_{i}$ denotes discrimination of
item $i$, which is the slope at the inflection point, and $c_{i}$ again
describes the probability of (pseudo)-guessing. With $c_{i}=0$, model (1)
becomes a 2PL IRT model or a simple logistic regression model in the non-IRT
framework.

Ordinal models may be employed to account for ordinal responses in items, such
as those involved in psychological assessments. One such model is the
adjacent-category model, also known as the Generalized Partial Credit Model
(GPCM) within the IRT framework. In this model, the so-called adjacent-
categories logits, which represent the logarithms for the ratio of
probabilities for two successive scores, are assumed to have a linear form:

$\displaystyle\log\left(\frac{\pi_{pik}}{\pi_{pi(k-1)}}\right)=\log\left(\frac{\mathrm{P}(Y_{pi}=k|\theta_{p})}{\mathrm{P}(Y_{pi}=k-1|\theta_{p})}\right)=a_{i}(\theta_{p}-b_{ik}),\quad
k=1,\dots,K_{i}.$

Here $a_{i}$ represents a common slope parameter for item $i$, while $b_{ik}$
denotes category-specific difficulty parameters, i.e., the ability level
needed to transition from category $k-1$ to $k$. Consequently, this
formulation yields category probabilities in the following form:

$\displaystyle\begin{split}\pi_{pik}&=\frac{\exp\left(\sum_{l=1}^{k}a_{i}(\theta_{p}-b_{il})\right)}{1+\sum_{t=1}^{K_{i}}\exp\left(\sum_{l=1}^{t}a_{i}(\theta_{p}-b_{il})\right)},\quad
k=1,\dots,K_{i},\\\
\pi_{pi0}&=\frac{1}{1+\sum_{t=1}^{K_{i}}\exp\left(\sum_{l=1}^{t}a_{i}(\theta_{p}-b_{il})\right)}=1-\sum_{k=1}^{K_{i}}\pi_{pik}.\end{split}$
(2)

To describe the functioning of both the correct answer and all distractors
(incorrect options) in multiple-choice items, the Nominal Response Model (NRM)
may be of use (see Figure 2c.). Within this model, the logarithms for the
ratio of probabilities for a distractor and the correct option are assumed to
have a linear form:

$\displaystyle\log\left(\frac{\pi_{pik}}{\pi_{ip0}}\right)=a_{ik}(\theta_{p}-b_{ik}),\quad
k=1,\dots,K_{i},$ (3)

with $a_{ik}$ being category-specific slopes for item $i$, and $b_{ik}$ being
category-specific difficulty parameters, i.e., the ability needed to
transition from category $k\neq 0$ to the baseline category $k=0.$

In both frameworks, IRT and non-IRT, it is possible to account for different
item types (binary, ordinal, or nominal) and, therefore, employ item-specific
models. In the regression models, item models are typically fitted separately,
while in the IRT framework, the model is fitted simultaneously for all items,
bringing a more precise estimate of the latent ability.

Item parameters in logistic regression-based models (binary, ordinal, or
nominal) are typically estimated via the maximum likelihood method, using, for
example, the iteratively reweighted least squares algorithm. In extensions of
the logistic regression accounting for possible guessing or slipping,
algorithms allowing for box constraints may be considered, such as "L-BFGS-B"
by [64]. Other possibilities, which seem more stable and precise, include the
EM algorithm or method based on the parametric link function [76]. Item
parameters of IRT models are often estimated using the marginal maximum
likelihood method with the EM algorithm. Person parameters may be estimated
using several methods based on either maximum likelihood or Bayesian
approaches; see, for example, implementations in the mirt package [65].

####  2.2.2 Differential item functioning

Differential item functioning (DIF) describes a situation when two respondents
with the same level of latent trait but from different groups have different
probabilities of answering an item correctly. These differences in
probabilities between groups may be caused by a secondary latent trait
unintentionally measured by the measurement instrument. In such a case, DIF
points to potentially unfair items. However, not all DIF items are necessarily
unfair. In all cases, DIF detection may provide a deeper understanding of the
strengths and weaknesses of tested subgroups.

There are a number of methods that can be used for the task of DIF detection.
Traditional approaches, which are implemented in the difR package [80],
include, for example, the delta method [59], the Mantel-Haenszel test [84], or
SIBTEST [103]. The logistic regression method [105] incorporates the effect
$\beta_{2i}$ of the group membership variable and its interaction $\beta_{i3}$
with the matching criterion (observed trait variable) $\theta_{p}$:

$\displaystyle\pi_{pi}=\mathrm{P}(Y_{pi}=1|\theta_{p},G_{p})=\frac{1}{1+\exp\left(-\beta_{0i}-\beta_{1i}\theta_{p}-\beta_{2i}G_{p}-\beta_{3i}G_{p}\theta_{p}\right)}.$
(4)

This group-specific model allows for testing the significance of the group
effects $\beta_{2i}$ and $\beta_{3i}$ using, for example, the likelihood ratio
test or Wald test of the submodel. Analogously to the logistic regression for
a description of item functioning, this approach can be extended to account
for possible guessing or slipping when answering and even to let these item
characteristics vary between groups [69]. Implementation of these models is
available in the difNLR package [75].

Another branch of DIF detection methods includes IRT models. In contrast to
regression-based approaches, IRT models for DIF detection are typically fitted
separately for the two groups, and item parameters are subsequently scaled.
Again, a test of Wald type [77, in the IRT framework, this is called Lord’s
test;] can be used to test differences between the two groups; implementation
is offered in the mirt package [65]. Alternatively, Raju’s test [96, 97] was
developed to test whether the area between the two estimated probability
curves is significantly different from zero; implementation is offered in the
difR package [80]. The group-specific IRT models once again estimate latent
traits together with item parameters, which, despite being computationally
more demanding, offer higher precision in both estimates. In contrast, the
regression-based methods provide a possibility to use matching criteria beyond
the total test score, its standardized version, or a variable describing the
measured trait (see Section 4).

###  2.3 Computerized adaptive testing

Computerized Adaptive Testing (CAT) employs IRT modeling within an iterative
algorithm to provide an efficient measurement [83]. Following the respondent’s
answer to an item, their ability estimate is updated. Subsequently, the next
item is selected as the most informative and suitable item for that ability
level from a pool of pre-calibrated items. This process is repeated until the
termination criteria are met, which may include factors such as the precision
of the ability estimate, the number of administered items, or time (see Figure
3).

Select and display the first item(s) Collect respondent’s answer Estimate
respondent’s level of ability Select and display the next item Are the
termination criteria met? End and return test results YesNo Figure 3:
Flowchart of CAT.

One approach to analyze the settings of a CAT, such as optimal item selection
criterion or termination criteria, involves conducting a post-hoc analysis on
the fixed test that has already been administered. This post-hoc analysis
utilizes specified CAT settings along with responses from selected respondents
to simulate the test in an adaptive environment. The implementation of this
process is provided by the mirtCAT package [66] or the catR package [82, 81].

###  2.4 More advanced psychometric topics

As the data complexity rises and computational resources become increasingly
abundant, new, more complex methods are being developed. One area of
computational psychometrics [107] involves the analysis of item wording in
multi-item measurements. Text analysis and natural language processing are
also involved in automatic item generation or in the rating of essays. To
support item development and to improve understanding of item cognitive
demands, [104] extracted various text features from item text wording and
trained machine learning algorithms to estimate item difficulty.

Measurement models, such as the IRT models, may be encompassed in more
intricate Structural Equation Models (SEM), which can analyze the
relationships and causal inferences among variables, some of which may be
latent. These sophisticated analyses are available in the lavaan [101], or the
blavaan [91, 90] packages. When dealing with multiple test forms, the _test
equating_ is necessary to obtain comparable test scores, which can be
performed with the equateIRT [63] or the kequate [58] packages. All of the
above are signs of the rapid evolution of the field, and they pose an argument
or motivation to create new modules that bring the topics to a broader
audience.

##  3 Extending ShinyItemAnalysis with add-on modules

The SIA application is developed in R [95] leveraging the shiny package [67],
complemented by direct implementation in HTML, CSS, and JavaScript for certain
components. The application is part of the ShinyItemAnalysis package and can
easily be started via the run_app() function. In the newest version, the SIA
application automatically detects additional add-on packages available at the
official repository and have not yet been installed:

⬇

R> library(ShinyItemAnalysis)

This is ShinyItemAnalysis version 1.5.1

- to run the interactive {shiny} app, call ‘run_app()‘

- to learn more, visit ‘ShinyItemAnalysis.org‘

R> run_app()

Additional SIA Modules are available! Do you want to install any of these?

1: All

2: None

3: EduTestItemAnalysis

4: EduTestTextAnalysis

5: IRR2FPR

6: SIAmodules

In the rest of this section, we explain in more detail how these modules can
be built and how they interact with the main SIA application.

###  3.1 SIA modules architecture

To extend the current application, we stay within the boundaries of R and
shiny. The choice of this widely known framework is deliberate as many R users
and developers are already familiar with it, and there is no need to learn any
new programming language to create an SIA module (compared to some other
software providing GUI extensions, mentioned in Section 1).

The SIA modules are implemented as standard shiny modules, which are a set of
two functions – one for the server logic and the other for the user interface
(UI). We leverage the fact that a shiny module is a well-delineated part of
the interactive application that can be bundled as a set of two callable
functions [108]. This may differ from a typical standalone shiny application,
which is defined and served from a single app.R file or a pair of ui.R and
server.R files.

The SIA modules are distributed in ordinary R packages. Developers can create
a new package with one or more modules or "append" the modules to their
existing packages. Modules available for a package are described in a YAML
file, which stores some metadata (title and category) and, crucially, provides
bindings to the aforementioned module’s functions.

###  3.2 Module discovery and usage

The SIA modules are intended for integration into the SIA application. Upon
launching the application, a mechanism is dispatched to locate SIA modules
within R packages installed in the user’s library. Initially, the SIA
application collects a list of packages claiming they contain SIA modules.
This is declared in the DESCRIPTION file of each module package with

⬇

Config/ShinyItemAnalysis/module: true

The following code is used in the inst/Modules.R file of the ShinyItemAnalysis
package to collect the names of available packages containing SIA modules:

⬇

desc_field <- "Config/ShinyItemAnalysis/module"

names(

which(

utils::installed.packages(fields = desc_field)[, desc_field] == "true"

)

)

All such "SIA module" packages are loaded and attached using the standard
library() call to ensure that any nontrivial features, such as compiled code
or S3/S4 methods, work as usual. A reference to the namespace environment of
the currently loaded module package is kept in the ns object. Subsequently,
the YAML file of each module package is searched for the modules’ metadata and
function bindings. The server function of each module is then located within
the package’s namespace environment and invoked using do.call() with the
module’s unique identifier as the first argument and with a list of SIA’s
reactive, reactiveVal, and reactiveValues as the second argument. This enables
the module to reuse any reactive object present in the parent application (see
Section 3.3 for more details):

⬇

do.call(ns[[mod_desc$binding$server]], list(id = mod_id, server_dots))

The UI of the module function is called in a similar fashion, but inside
shiny’s appendTab() function that appends a new entry to the correct menu:

⬇

appendTab(

inputId = "navbar",

menuName = mod_desc$category,

tab = tabPanel(

title = mod_desc$title, value = mod_id,

do.call(ns[[mod_desc$binding$ui]], list(id = mod_id, ui_dots))

)

)

In the code snippet above, ns refers to the package’s namespace environment,
mod_desc is a list containing the module’s metadata, and mod_id is the unique
module identifier created by concatenating the package’s and the module’s
names (to prevent name collisions).

The category specified in mod_desc$category within the module’s metadata is
used to place the module tab in the desired menu section of the application.
Currently, available categories include "Scores", "Validity", "Reliability",
"Item analysis", "Regression", "IRT models", and "DIF/Fairness". Any other
value assigned to the category attribute will automatically position the
module into the "Modules" section of the application.

###  3.3 Module development with SIAtools

In order to streamline the development of new SIA modules, we introduce a
companion package called SIAtools [94]. This package comprises a collection of
functions for constructing and managing modules, along with ready-to-use
templates, guides, and various tests to ensure a smooth integration of the
module into the SIA application. It is important to note that developers still
retain the option to create SIA modules from scratch, as outlined in previous
sections. However, with the assistance of SIAtools, developers can focus
solely on the content of the module itself.

Developers interested in creating a SIA module may find themselves in two
distinct situations: (1) they possess an existing R package they wish to
extend with one or more SIA modules, or (2) they aim to develop a standalone
SIA module. In the latter scenario, the SIAtools package offers the capability
to create an R package serving as a "container" for the SIA module. This can
be achieved either via the R console or through the GUI wizard provided by
RStudio. In both cases, developers can utilize the add_module() function to
configure the package to be compatible with the SIA application. This function
automatically generates an entry in the YAML file and creates a template
tailored for a new SIA module. Subsequently, developers are encouraged to fill
the template with their code, preview the work in progress with the
preview_module() function, and iterate until satisfied with the outcome.

In the following example, we demonstrate the process of extending the SIA
application with the SIA module using the companion SIAtools package. We begin
by installing the package from CRAN and then loading it:

⬇

install.packages("SIAtools")

library("SIAtools")

To create a SIA module from scratch, one may call
create_module_project("new_proj")111The final package is provided as
SampleModulePackage_0.1.0.tar.gz in Supplementary Materials. (where new_proj
represents the name of the new RStudio project that will be created in the
current working directory). Another way is to use the RStudio project wizard
(File > New project > New Directory > ShinyItemAnalysis Module Project). Upon
project creation, a welcome message containing basic instructions will pop up.

For the illustration, we will build a simplified version of the CAT module
from the SIAmodules package (the resulting simplified module is depicted in
Figure 4; the "reference" module is described and shown in Section 4.2).
Initially, we call add_module("cat") to create a new module within the
project. This opens the YAML file (also called "SIA Modules Manifest" in the
SIAtools package), which contains the module specification. Within this file,
we modify the title to "CAT Example" and define the module’s category (in this
instance, "Modules", see Section 3.2). Following these adjustments, the YAML
file should resemble the following:

⬇

sm_cat:

title: CAT Example

category: Modules

binding:

ui: sm_cat_ui

server: sm_cat_server

The automatically generated module ID (sm_cat), located in the first row,
serves as the identifier for the module throughout the application.
Additionally, the function bindings, which direct to the module’s functions,
are pre-filled. These functions are housed in sm_cat.R, which is automatically
created and opened for the developer in RStudio alongside the edited YAML
file. In the sm_cat.R source file, function skeletons are provided with useful
comments and recommended structure, along with placeholder texts indicating
where to insert the code for both the server logic and UI components.

In the first part of the sm_cat.R source file, developers can furnish user-
facing documentation, which will be listed together with other functions
potentially exported by the package (see ?SIAmodules::sm_cat for an example of
the documentation for CAT module living in the SIAmodules package). For
instance, the documentation for the simplified example is:

⬇

# Module documentation \----------------------------------------------------

# This is the user-facing documentation.

#’ CAT Example

#’

#’ This SIA module demonstrates basic principles of module development on a

#’ simplified version of Computerized Adaptive Tests module from ‘SIAmodules‘

#’ package.

#’

#’

#’ @author

#’ John Doe

#’

#’ @references

#’ Doe, J. (3024). Sample module. *Journal of Sample Modules, 1*(1), 1--10.

#’

#’ @name sm_cat

#’ @family SIAmodules

#’

NULL

This documentation offers a brief overview of the module’s purpose and
functionality, enhancing clarity for users accessing the package.

The second part of the sm_cat.R file is dedicated to defining UI function,
where developers declare all the components and layout using functions
provided by the shiny package.

⬇

## UI part \----------------------------------------------------------------

#’ ‘sm_cat‘ module (internal documentation)

#’

#’ [code removed for brevity]

#’

#’ @param id *character*, the ID assigned by ShinyItemAnalysis. **Do not set

#’ any default value!**

#’ @param imports *list* of reactive objects exported by ShinyItemAnalysis.

#’ See ‘vignette("imports", "SIAtools")‘ for more details on how to use

#’ objects from the ShinyItemAnalysis app.

#’ @param ... additional parameters (not used at the moment).

#’

#’ @keywords internal

#’ @rdname sm_cat_internal

#’

#’ @import shiny

#’

sm_cat_ui <- function(id, imports = NULL, ...) {

ns <- NS(id) # shorthand for NS(id, <inputId>)

# Any ‘inputId‘ and ‘outputId‘ of {shiny} UI elements MUST be "wrapped" in

# ‘ns()‘ call! Use ‘SIAtools::lint_ns()‘ to check for possible omissions.

tagList(

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

h3("Computerized Adaptive Tests"),

selectInput(ns("irt_model"),

"IRT model to use",

choices = c(

"Module’s example 2PL" = "example",

"SIA-fitted IRT for binary data" = "sia_binary"

)

),

sliderInput(

ns("true_theta"),

"Respondent’s true ability",

value = 1, min = -4, max = 4, step = .1

),

plotOutput(ns("plot"))

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

)

}

In the code snippet provided above, inside the tagList() function, we included
a level 3 heading title, an input element used for the selection of an IRT
model (further details on this input will be provided later); a slider input
for adjusting the true ability of a respondent, and the last line refers to
the single output of this sample module, which is a plot.

An important detail to notice is the usage of the ns()222The name of the ns()
function is derived from the term ”namespace”. However, it does not refer to R
package namespace as mentioned in Section 3.2. function around all shiny input
and output IDs within the UI function. This function is defined directly in
the sm_cat.R source file and ensures that UI input and output IDs match with
those expected by their server logic counterparts. The SIAtools package is
shipped with a simple linter compatible with the lintr package [74] that
checks for the omission of this "namespacing" practice, which can be
challenging to debug (for further details, refer to ?module_namespace_linter).

In the third part of the sm_cat.R file, we define the server logic, which, in
our case, is done within the sm_cat_server() function.

⬇

## Server part \------------------------------------------------------------

#’ @rdname sm_cat_internal

#’

#’ @importFrom mirtCAT generate_pattern mirtCAT

#’

sm_cat_server <- function(id, imports = NULL, ...) {

moduleServer(id, function(input, output, session) {

ns <- session$ns

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

mod <- reactive({

switch(input$irt_model,

example = example_2pl_mod,

sia_binary = imports$IRT_binary_model()

)

})

sim_res <- reactive({

response_pattern <- generate_pattern(mod(), input$true_theta)

mirtCAT(

mo = mod(), local_pattern = response_pattern,

start_item = "MI", criteria = "MI", design = list(min_SEM = .4)

)

})

output$plot <- renderPlot({

plot(sim_res(), SE = qnorm(.975))

})

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

})

}

In the server logic code, we create a reactive expression mod() that depends
on the model selected in the corresponding input. By default, it retrieves an
example IRT model, previously defined and stored as internal data of the
package (see the complete code in the supplementary materials). This allows us
to reference the model within our package. The second reactive expression is
the output from the simulation of CAT administration itself, based on the
response pattern generated using the true latent ability selected with the
slider and the IRT model in use. In each step of the iterative CAT algorithm
(Figure 3), the simulation will "present" the item with maximal information
(MI) for a "current" estimate of the latent ability, and the test will stop
when a minimal standard error of 0.40 is reached. In the final segment of the
provided code snippet, we call the plot() method of the mirtCAT package [66]
to generate a plot of the post-hoc simulation. It is worth noting that every
function from external packages must be imported using, for instance,
@importFrom tag and declared in the DESCRIPTION file because we are
constructing a formal R package [109].

To preview work in progress, simply call the preview_module() function. By
default, all roxygen2 [110] blocks333A sequence of lines starting with #’.
automatically convert into .Rd R manual files and NAMESPACE entries. The
entire package is loaded and attached without requiring any installation in
order to streamline the development as much as possible. The module’s
functions are called within a simple shiny application skeleton (see
?preview_module for more details), and upon execution, the application
launches (see Figure 4).

Figure 4: A preview of sample CAT module

The module, by default, utilizes the aforementioned example IRT model (Figure
4). However, it is important to note that an error message would be displayed
if we selected another model from the SIA application. This occurs because,
referring to the source code of the module’s server logic, the module attempts
to utilize the imports$IRT_binary_model() reactive, which does not exist. This
discrepancy arises from the absence of a connection to the SIA application in
the preview, resulting in the imports argument of the sm_cat_server() function
being NULL444Nevertheless, preview_module() function allows to pass inputs in
preview mode, see the documentation..

To test the module fully connected to the parent SIA application, we have to
build the package from the source and install it as usual. Then, once we run
the application locally with ShinyItemAnalysis::run_app(), our module will
appear in the Modules tab (as specified in the YAML file), and we can test it
with the IRT model fitted by the SIA application. Now, the module receives the
imports list of reactives populated by the running SIA application and is able
to utilize an IRT model from the SIA application, as we will demonstrate in
the next Section.

###  3.4 Turning a standalone shiny application into a SIA module

A standalone shiny application is typically developed using separate ui.R and
server.R files, which may not necessarily be housed within a package
structure555If so, they are usually located in inst directory ”as is”.. As
previously outlined in Section 3.1, this approach differs substantially from
the approach taken by SIA modules and several other packages related to shiny
application development, such as golem [71] or rhino [112]. In order to
migrate from a standalone application to an SIA module, the following
essential steps are required:

  1. 1.

Create an R package, or use an existing one. In the DESCRIPTION file, include
a line containing Config/ShinyItemAnalysis/module field set to true.

  2. 2.

Within the R directory of the package, create a new R source file (e.g.,
sm_cat.R), which includes the module documentation, the UI function, and the
server logic function. Generally, it is possible to copy-paste the existing
code from the original ui.R and server.R files. However, ensure that every
input and output ID in the UI portion is enclosed in ns(). There is a
convenient check provided by the lint_ns() function of the SIAtools package.

  3. 3.

Create a YAML file that provides the title, category, and function bindings.
For the exact structure, refer to Section 3.3.

###  3.5 Module distribution

As outlined earlier, SIA modules are distributed within standard R packages
(containing one or more modules). The official repository is located at
https://shinyitemanalysis.org/repo/. Users can install these packages using
the conventional install.packages() function and this URL provided in repos
argument. Since the repository hosts only the packages of interest without any
external dependencies, users need to provide their usual Comprehensive R
Archive Network (CRAN) mirror to ensure the proper installation of the module
packages. For instance, to install the SIA module contained in the
EduTestItemAnalysis package [93], which is not available from the CRAN, the
following code can be used:

⬇

install.packages("EduTestItemAnalysis", repos =
c("https://shinyitemanalysis.org/repo/", "https://cloud.r-project.org"))

To retrieve the names of already available module packages, users or
developers can employ the following code:

⬇

rownames(available.packages(repos = "https://shinyitemanalysis.org/repo/"))

The developers are supposed to build their source packages and submit them to
the first author of the paper via e-mail.

##  4 Sample SIA modules

The sample SIA modules presented in this section may serve as an inspiration
for the possible extensions of the main SIA application. Some of these modules
use their datasets only, while others allow interaction with data from the
main SIA application or even generate datasets to be passed into the main SIA
application.

###  4.1 EduTestItemAnalysis: Tailoring data upload, adding data complexity

The EduTestItemAnalysis module within the eponymous EduTestItemAnalysis
package [93] is specifically tailored to accommodate the format of publicly
available Czech Matura Exam data. It serves as an example of how to create a
customized data upload procedure that diverges from the main SIA application.
Notably, the EduTestItemAnalysis module enables users to upload test metadata
specifying the type of each item, i.e., some items may follow the 3PL IRT
model (1) or its restricted 2PL version, some may be modeled by the GPCM
defined in Equation (2), and some by the NRM defined in Equation (3), wherein
the correct response must also be specified. This demonstrates the capability
to extend the IRT analysis within the SIA application which currently supports
only the tests with a single item type. The item-specific IRT modeling is
provided in the respective tab of the module, alongside some customized
traditional item analyses. Additionally, the module offers the functionality
to create binary grouping and/or criterion variables from a factor variable
with multiple levels, which can be utilized for DIF detection within the SIA
application (Figure 5).

Figure 5: Custom dataset editing in the EduTestItemAnalysis module to align
with the SIA application format

This demonstrates the capability to extend the IRT analysis within the SIA
application, which currently supports only the tests with a single item type.
The item-specific IRT modeling is provided in the respective tab of the
module, alongside some customized traditional item analyses. Additionally, the
module offers the functionality to create binary grouping and/or criterion
variables from a factor variable with multiple levels, which can be utilized
for DIF detection within the SIA application.

The EduTestItemAnalysis module also serves as a non-trivial case study
showcasing the module-to-application communication. Upon uploading a dataset
to the module, users have the opportunity to modify it to align with the SIA
application and reuse it in other tabs beyond the module’s scope. In order to
be able to run all the analyses of the SIA application and its additional
modules, users are offered to pass data uploaded and edited in the
EduTestItemAnalysis module directly to the main application via the "Pass data
to SIA" button (see Figure 5).

###  4.2 CAT module: Utilizing models across the application and its modules

The CAT module (Figure 6) from the SIAmodules package represents a fully
refined version of the simplified sample module that we used for the in-depth
demonstration described in Section 3. This module offers the possibility to
use the NRM (3) or other IRT models fitted within the main SIA
application666Note that there is no need to fit the models in the ”IRT models”
tab beforehand. As soon as the reactive expression used by our module is
invoked, the selected model is fitted with the defaults provided in the UI of
the respective tab. When the underlying data or any relevant inputs change,
the model automatically refits., possibly on data uploaded by the user. The
module performs a CAT post-hoc analysis, i.e., a simulation of the
respondent’s performance under an adaptive test of pre-defined settings.

Figure 6: CAT module using the data uploaded in the EduTestItemAnalysis module
and utilizing the IRT model fitted by the main SIA application

###  4.3 DIF-C: Extending the DIF analysis to longitudinal setting

Regression-based methods for DIF detection offer the flexibility to
incorporate external matching criteria, such as pre-test scores. Consequently,
these models can be used to detect so-called DIF in change [88, DIF-C;]. For
instance, in the just mentioned study focusing on learning competencies,
despite no discernible difference in total scores between basic and academic
school tracks in the 6th grade (Figure 7a.) and similarly in the 9th grade
(Figure 7b.), certain items still exhibited differential functioning between
these tracks in the 9th grade accounting for the prior knowledge of learning
competencies of students in the 6th grade (Figure 8) using logistic regression
model for DIF detection (4).

a. Histogram of total score in the 6th grade

b. Histogram of total score in the 9th grade

Figure 7: Illustration of differential item functioning in change

The DIF-C module from the SIAmodules package opens up the core analysis in an
interactive and directly reproducible way. Additionally, The corresponding
LearningToLearn dataset is conveniently accessible within the main application
and, therefore, can be analyzed using various psychometric models and
approaches. Additionally, DIF detection methods in the "DIF/Fairness" tab
offer the possibility to employ an observed score variable as the matching
criterion, which is the score from the 6th grade for the LearningToLearn. This
enables DIF-C analysis for this dataset. Moreover, DIF-C detection to other
datasets is achievable by providing the "Observed score" variable in the
"Data" section. While the DIF-C detection is accessible in the main
application, the DIF-C module from the SIAmodules package opens up the core
analysis of the paper [88] in an interactive and directly reproducible way. It
provides a step-by-step examination of both scores, a summary of the DIF-C
analysis, and plots of ICCs for individual items.

Figure 8: DIF-C module from the SIAmodules package

###  4.4 Inter-rater reliability: Analyzing ratings from multiple raters

Another aspect of data complexity not currently addressed in the main SIA
application involves ratings from multiple raters. When multiple raters are
involved, the assessment of inter-rater reliability (IRR) becomes pertinent,
typically analyzed through methods such as variance analysis or, more
generally, variance component models [86].

The IRR module within the SIAmodules package provides an interactive
demonstration of the issues of using IRR in restricted-range samples in the
context of grant proposal peer review. The module demonstrates that when
subsets of restricted-quality proposals are used, this will likely result in
zero estimates of IRR under many scenarios, although the global IRR may be
sufficient [70].

Figure 9: IRR module from the SIAmodules package

As another example of a module residing in the "Reliability" tab of the main
SIA application, the IRR2FPR module of the IRR2FPR package [61] provides an
interactive illustration of the calculation of binary classification metrics
from IRR [62].

###  4.5 EduTestTextAnalysis: Employing large models and compiled code

EduTestTextAnalysis module from the EduTestTextAnalysis package [92] seeks to
provide a tool for item difficulty prediction based solely on the item wording
[104, see]. The module does not use any data from the main application, nor
does it upload any tabular data. Instead, it uses text input fields and the
database of several item examples. This is another demonstration of the
versatility of the SIA modules pertaining to the input nature.

Another important feature that this module illustrates is the usage of complex
and large models spanning gigabytes of binary data. One of the crucial
independent variables in the predictive model is the cosine similarity of
different item wording parts [104], calculated employing the word2vec [111]
word embeddings model. In the module, we implemented a mechanism that can
download and cache the compressed binary model from the internet on demand and
utilize it immediately in the analysis, thus proving that large and complex
models are manageable in the proposed modular architecture. The
EduTestTextAnalysis module also demonstrates the use of compiled C++ libraries
that the word2vec package is wrapping.

Figure 10: EduTestTextAnalysis module from the EduTestTextAnalysis package

##  5 Summary and discussion

In this paper, we have outlined the process of developing new add-on modules
for the SIA interactive app in R with the help of the SIAtools package. We
demonstrated the fundamental principles and options using several modules that
are already available and of varying complexity. Interactive Shiny
applications have the potential to expand the user community, and when R code
is provided, they also help newcomers in adopting R. The platform we
introduced for add-on packages, utilized in the SIA framework, may ease
external collaboration and customization of shiny projects in general.

The main benefit of the add-on modules lies in their ability to enhance the
extensive capabilities and functionalities of the ShinyItemAnalysis package,
namely with contemporaneous methods and models or didactic showcases. This
package offers numerous features, including toy datasets, the option to upload
custom datasets, a variety of functions utilized in the interactive
application, and the possibility to generate automatic reports, among others.
As we have demonstrated with the CAT example module, several modules of the
SIAmodules package, and with the EduTestItemAnalysis and EduTestTextAnalysis
modules, SIA add-on modules can leverage datasets provided by the SIA
application but also use their own ones, up to large complex data processed
with efficient C++ libraries. Furthermore, they can extend beyond the data
types offered in the main SIA application. These modules can seamlessly
integrate with the main application; for instance, as we have demonstrated,
one module may pass data to the main application for analysis and subsequently
transfer the resulting model to another add-on module for further analysis and
interactive presentation of the results (e.g., post-hoc analysis of adaptive
test), and it does all of this automatically under the hood thanks to
reactivity principles of shiny.

To the best of our knowledge, only jamovi [106] and JASP [79] make their
analyses fully or partially available to be used programmatically from within
R console as they build upon the underlying R packages (jmv package [102] for
jamovi and various packages for JASP), although both are meant to be run as a
standalone software with GUI in the first place.

We offer the SIAtools package as a resource to facilitate SIA module
development. A similar toolkit is provided in the jmvtools package [78], which
also provides a few templates and crucially uses a proprietary compiler to
create a module that can be used in the jamovi software with GUI. On top of
that, jmvtools needs a special JavaScript runtime environment to operate. This
is also the case for another similar package called jaspTools [68], which
relies on a number of external dependencies as well. In contrast, the SIAtools
works solely within the confines of the R language.

There are several aspects worth considering in future versions of the
ShinyItemAnalysis, SIAtools, and SIAmodules packages. First, the SIA module
installation may, in the near future, be offered directly from the GUI of the
main SIA application. This will ease the module installation for those users
who are new to R. At present, we only provide a simple command-line interface
that offers the installation of currently available SIA modules on the
official repository. Second, the SIAtools package may become more refined in
terms of module testing, building, and submitting. Last but not least, the SIA
application offers the automatic generation of PDF and HTML reports. When a
module uses the data from the main SIA application, including module results
in the report may be feasible. Further automation of generating the reports
via a command-line environment may foster automation, reproducibility and
reuse with other packages.

While these improvements would increase the usefulness of the current
approach, the presented version already represents a valuable extension of the
ShinyItemAnalysis package. We believe that its innovative nature and practical
utility have the potential to not only inspire but also influence future
projects in this domain.

## Acknowledgments

The study was funded by the Czech Science Foundation project "Theoretical
Foundations of Computational Psychometrics" grant number 21-03658S, and by the
project "Research of Excellence on Digital Technologies and Wellbeing
CZ.02.01.01/00/22_008/0004583" which is co-financed by the European Union. The
authors would like to acknowledge František Bartoš for helpful comments on the
previous version of the manuscript.

## Online Supplementary Material

Supplementary material, including the accompanying R scripts, is available at
https://osf.io/cnbrh/.

## References

  * [1] American Educational Research Association (AERA), American Psychological Association (APA), National Council on Measurement in Education (NCME)  “Standards for educational and psychological testing”  American Educational Research Association, 2014
  * [2] Björn Andersson, Kenny Bränberg and Marie Wiberg  “Performing the kernel method of test equating with the package kequate”  In _Journal of Statistical Software_ 55.6, 2013, pp. 1–25  URL: https://doi.org/10.18637/jss.v055.i06
  * [3] William H Angoff and Susan F Ford  “Item-race interaction on a test of scholastic aptitude”  In _Journal of Educational Measurement_ 10.2 Wiley Online Library, 1973, pp. 95–105  DOI: 10.1111/j.1745-3984.1973.tb00787.x
  * [4] David J Bartholomew, Fiona Steele and Irini Moustaki  “Analysis of multivariate social science data”  CRC press, 2008
  * [5] František Bartoš  “IRR2FPR: Computing false positive rate from inter-rater reliability” R package version 0.1, 2024  URL: https://CRAN.R-project.org/package=IRR2FPR
  * [6] František Bartoš and Patrícia Martinková  “Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability”  In _British Journal of Mathematical and Statistical Psychology_ , 2024  DOI: 10.1111/bmsp.12343
  * [7] Michela Battauz  “equateIRT: An R package for IRT test equating”  In _Journal of Statistical Software_ 68.7, 2015, pp. 1–22  DOI: 10.18637/jss.v068.i07
  * [8] Richard H Byrd, Peihuang Lu, Jorge Nocedal and Ciyou Zhu  “A limited memory algorithm for bound constrained optimization”  In _SIAM Journal on Scientific Computing_ 16.5 SIAM, 1995, pp. 1190–1208  DOI: 10.1137/0916069
  * [9] R. Chalmers  “mirt: a multidimensional item response theory package for the R environment”  In _Journal of Statistical Software_ 48.6, 2012, pp. 1–29  DOI: 10.18637/jss.v048.i06
  * [10] R. Chalmers  “Generating adaptive and non-adaptive test interfaces for multidimensional item response theory applications”  In _Journal of Statistical Software_ 71.5, 2016, pp. 1–39  DOI: 10.18637/jss.v071.i05
  * [11] Winston Chang et al.  “shiny: Web application framework for R” R package version 1.8.1.1, 2024  URL: https://CRAN.R-project.org/package=shiny
  * [12] Tim Jong and Don Bergh  “jaspTools: Helps preview and debug JASP analyses” R package version 0.18.3.9000, 2024  URL: https://github.com/jasp-stats/jaspTools
  * [13] Adéla Drabinová and Patrícia Martinková  “Detection of differential item functioning with nonlinear regression: A non-IRT approach accounting for guessing”  In _Journal of Educational Measurement_ 54.4 Wiley Online Library, 2017, pp. 498–517  DOI: 10.1111/jedm.12158
  * [14] Elena A Erosheva, Patrícia Martinková and Carole J Lee  “When zero may not be zero: a cautionary note on the use of inter-rater reliability in evaluating grant peer review”  In _Journal of the Royal Statistical Society: Series A (Statistics in Society)_ 184.3, 2021, pp. 904–919  DOI: 10.1111/rssa.12681
  * [15] Colin Fay, Vincent Guyader, Sébastien Rochette and Cervan Girard  “golem: a framework for robust Shiny applications” R package version 0.4.1, 2023  URL: https://CRAN.R-project.org/package=golem
  * [16] Ian Fellows  “Deducer: a data analysis GUI for R”  In _Journal of Statistical Software_ 49.8, 2012, pp. 1–15  DOI: 10.18637/jss.v049.i08
  * [17] John Fox  “The R commander: a basic-statistics graphical user interface to R”  In _Journal of Statistical Software_ 14.9, 2005, pp. 1–42  DOI: 10.18637/jss.v014.i09
  * [18] Jim Hester et al.  “lintr: a ’linter’ for R code” R package version 3.1.1, 2023  URL: https://github.com/r-lib/lintr
  * [19] Adéla Hladká and Patrícia Martinková  “difNLR: Generalized logistic regression models for DIF and DDF detection”  In _The R Journal_ 12.1, 2020, pp. 300–323  DOI: 10.32614/RJ-2020-014
  * [20] Adéla Hladká, Patrícia Martinková and Marek Brabec  “Parameter estimation in generalised logistic model with application to DIF detection”  In _arXiv_ , 2023  DOI: 10.48550/arXiv.2302.12648
  * [21] Frederic M Lord  “Applications of item response theory to practical testing problems”  Milton Park, Abingdon, Oxon: Routledge, 1980  DOI: 10.4324/9780203056615
  * [22] Jonathon Love  “jmvtools: Tools to build jamovi modules” R package version 2.5.1, 2024  URL: https://github.com/jamovi/jmvtools
  * [23] Jonathon Love et al.  “JASP: Graphical statistical software for common statistical designs”  In _Journal of Statistical Software_ 88.2, 2019, pp. 1–17  DOI: 10.18637/jss.v088.i02
  * [24] D. Magis, S. Beland, F. Tuerlinckx and P. De Boeck  “A general framework and an R package for the detection of dichotomous differential item functioning”  In _Behavior Research Methods_ 42, 2010, pp. 847–862  DOI: 10.3758/brm.42.3.847
  * [25] David Magis and Juan Ramon Barrada  “Computerized adaptive testing with R: Recent updates of the package catR”  In _Journal of Statistical Software, Code Snippets_ 76.1, 2017, pp. 1–19  DOI: 10.18637/jss.v076.c01
  * [26] David Magis and Gilles Raîche  “Random generation of response patterns under computerized adaptive testing with the R package catR”  In _Journal of Statistical Software_ 48.8, 2012, pp. 1–31  DOI: 10.18637/jss.v048.i08
  * [27] David Magis, Duanli Yan and Alina A Von Davier  “Computerized adaptive and multistage testing with R: Using packages catR and mstR”  Springer, 2017  DOI: 10.1007/978-3-319-69218-0
  * [28] Nathan Mantel and William Haenszel  “Statistical aspects of the analysis of data from retrospective studies”  In _Journal of the National Cancer Institute_ 22.4, 1959, pp. 719–748  DOI: 10.1093/jnci/22.4.719
  * [29] Patricia Martinková and Adéla Hladká  “Computational aspects of psychometric methods: With R”  Chapman and Hall/CRC, 2023  DOI: 10.1201/9781003054313
  * [30] Patrícia Martinková and Adéla Drabinová  “ShinyItemAnalysis for teaching psychometrics and to enforce routine analysis of educational tests”  In _The R Journal_ 10.2, 2018, pp. 503–515  DOI: 10.32614/RJ-2018-074
  * [31] Patrícia Martinková and Jan Netík  “SIAmodules: Modules for ’ShinyItemAnalysis”’ R package version 0.1.1, 2024  URL: https://cran.r-project.org/package=SIAmodules
  * [32] Patrícia Martinková, Adéla Hladká and Eva Potužníková  “Is academic tracking related to gains in learning competence? Using propensity score matching and differential item change functioning analysis for better understanding of tracking implications”  In _Learning and Instruction_ 66 Elsevier, 2020, pp. 101286  DOI: 10.1016/j.learninstruc.2019.101286
  * [33] Patrícia Martinková, František Bartoš and Marek Brabec  “Assessing inter-rater reliability with heterogeneous variance components models: Flexible approach accounting for contextual variables”  In _Journal of Educational and Behavioral Statistics_ 48.3, 2023, pp. 349–383  DOI: 10.3102/10769986221150517
  * [34] Edgar C. Merkle and Yves Rosseel  “blavaan: Bayesian structural equation models via parameter expansion”  In _Journal of Statistical Software_ 85.4, 2018, pp. 1–30  DOI: 10.18637/jss.v085.i04
  * [35] Edgar C. Merkle, Ellen Fitzsimmons, James Uanhoro and Ben Goodrich  “Efficient Bayesian structural equation modeling in Stan”  In _Journal of Statistical Software_ 100.6, 2021, pp. 1–22  DOI: 10.18637/jss.v100.i06
  * [36] Jan Netík and Patrícia Martinková  “EduTestItemAnalysis: Tools to support psychometric analyses at ’CERMAT”’ R package version 0.1.4, 2024  URL: https://www.ShinyItemAnalysis.org
  * [37] Jan Netík and Patrícia Martinková  “SIAtools: "ShinyItemAnalysis" modules development toolkit” R package version 0.1.1, 2024  URL: https://cran.r-project.org/package=SIAtools
  * [38] Jan Netík, Jana Dlouhá and Patrícia Martinková  “EduTestTextAnalysis: Predicting multiple-choice item difficulty from text” R package version 0.1.9, 2024  URL: https://www.ShinyItemAnalysis.org
  * [39] R Core Team  “R: a language and environment for statistical computing”, 2024  R Foundation for Statistical Computing  URL: https://www.R-project.org/
  * [40] Nambury S Raju  “The area between two item characteristic curves”  In _Psychometrika_ 53.4 Springer, 1988, pp. 495–502  DOI: 10.1007/bf02294403
  * [41] Nambury S Raju  “Determining the significance of estimated signed and unsigned areas between two item response functions”  In _Applied Psychological Measurement_ 14.2 Sage Publications Sage CA: Thousand Oaks, CA, 1990, pp. 197–207  DOI: 10.1177/014662169001400208
  * [42] Calyampudi Radhakrishna Rao and Sandip Sinharay  “Psychometrics” 26, Handbook of statistics  Amsterdam; Boston: Elsevier, 2007
  * [43] William Revelle  “psych: Procedures for psychological, psychometric, and personality research” R package version 2.4.3  Evanston, Illinois: Northwestern University, 2024  URL: https://CRAN.R-project.org/package=psych
  * [44] Stefan Rödiger, Thomas Friedrichsmeier, Prasenjit Kapat and Meik Michalke  “RKWard: a comprehensive graphical user interface and integrated development environment for statistical analysis with R”  In _Journal of Statistical Software_ 49.9, 2012, pp. 1–34  DOI: 10.18637/jss.v049.i09
  * [45] Yves Rosseel  “lavaan: An R package for structural equation modeling”  In _Journal of Statistical Software_ 48.2, 2012, pp. 1–36  DOI: 10.18637/jss.v048.i02
  * [46] Ravi Selker, Jonathon Love and Damian Dropmann  “jmv: The ’jamovi’ analyses” R package version 2.4.11, 2023  URL: https://CRAN.R-project.org/package=jmv
  * [47] Robin Shealy and William Stout  “A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF”  In _Psychometrika_ 58.2 Springer, 1993, pp. 159–194  DOI: 10.1007/bf02294572
  * [48] Lubomír Štěpánek, Jana Dlouhá and Patrícia Martinková  “Item difficulty prediction using item text features: Comparison of predictive performance across machine-learning algorithms”  In _Mathematics_ 11.4104, 2023  DOI: 10.3390/math11194104
  * [49] Hariharan Swaminathan and H Jane Rogers  “Detecting differential item functioning using logistic regression procedures”  In _Journal of Educational Measurement_ 27.4, 1990, pp. 361–370  DOI: 10.1111/j.1745-3984.1990.tb00754.x
  * [50] The jamovi project  “jamovi” Computer software, version 2.5, 2024  URL: https://www.jamovi.org
  * [51] Alina A Davier, Robert J Mislevy and Jiangang Hao  “Computational psychometrics: New methodologies for a new generation of digital learning and assessment: With examples in R and python”  Springer Nature, 2022
  * [52] Hadley Wickham  “Mastering Shiny: Build interactive apps, reports, and dashboards power by R”  Sebastopol, CA: O’Reilly Media, 2021, pp. 259–286
  * [53] Hadley Wickham and Jenny Bryan  “R Packages: Organize, Test, Document, and Share Your Code”  Beijing Boston Farnham Sebastopol Tokyo: O’Reilly Media, 2023
  * [54] Hadley Wickham, Peter Danenberg, Gábor Csárdi and Manuel Eugster  “roxygen2: In-line documentation for R” R package version 7.3.1, 2024  URL: https://CRAN.R-project.org/package=roxygen2
  * [55] Jan Wijffels and Kohei Watanabe  “word2vec: Distributed representations of words” R package version 0.4.0, 2023  URL: https://CRAN.R-project.org/package=word2vec
  * [56] Kamil Żyła et al.  “rhino: a framework for enterprise Shiny applications” R package version 1.7.0, 2024  URL: https://CRAN.R-project.org/package=rhino

## References

  * [57] American Educational Research Association (AERA), American Psychological Association (APA), National Council on Measurement in Education (NCME)  “Standards for educational and psychological testing”  American Educational Research Association, 2014
  * [58] Björn Andersson, Kenny Bränberg and Marie Wiberg  “Performing the kernel method of test equating with the package kequate”  In _Journal of Statistical Software_ 55.6, 2013, pp. 1–25  URL: https://doi.org/10.18637/jss.v055.i06
  * [59] William H Angoff and Susan F Ford  “Item-race interaction on a test of scholastic aptitude”  In _Journal of Educational Measurement_ 10.2 Wiley Online Library, 1973, pp. 95–105  DOI: 10.1111/j.1745-3984.1973.tb00787.x
  * [60] David J Bartholomew, Fiona Steele and Irini Moustaki  “Analysis of multivariate social science data”  CRC press, 2008
  * [61] František Bartoš  “IRR2FPR: Computing false positive rate from inter-rater reliability” R package version 0.1, 2024  URL: https://CRAN.R-project.org/package=IRR2FPR
  * [62] František Bartoš and Patrícia Martinková  “Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability”  In _British Journal of Mathematical and Statistical Psychology_ , 2024  DOI: 10.1111/bmsp.12343
  * [63] Michela Battauz  “equateIRT: An R package for IRT test equating”  In _Journal of Statistical Software_ 68.7, 2015, pp. 1–22  DOI: 10.18637/jss.v068.i07
  * [64] Richard H Byrd, Peihuang Lu, Jorge Nocedal and Ciyou Zhu  “A limited memory algorithm for bound constrained optimization”  In _SIAM Journal on Scientific Computing_ 16.5 SIAM, 1995, pp. 1190–1208  DOI: 10.1137/0916069
  * [65] R. Chalmers  “mirt: a multidimensional item response theory package for the R environment”  In _Journal of Statistical Software_ 48.6, 2012, pp. 1–29  DOI: 10.18637/jss.v048.i06
  * [66] R. Chalmers  “Generating adaptive and non-adaptive test interfaces for multidimensional item response theory applications”  In _Journal of Statistical Software_ 71.5, 2016, pp. 1–39  DOI: 10.18637/jss.v071.i05
  * [67] Winston Chang et al.  “shiny: Web application framework for R” R package version 1.8.1.1, 2024  URL: https://CRAN.R-project.org/package=shiny
  * [68] Tim Jong and Don Bergh  “jaspTools: Helps preview and debug JASP analyses” R package version 0.18.3.9000, 2024  URL: https://github.com/jasp-stats/jaspTools
  * [69] Adéla Drabinová and Patrícia Martinková  “Detection of differential item functioning with nonlinear regression: A non-IRT approach accounting for guessing”  In _Journal of Educational Measurement_ 54.4 Wiley Online Library, 2017, pp. 498–517  DOI: 10.1111/jedm.12158
  * [70] Elena A Erosheva, Patrícia Martinková and Carole J Lee  “When zero may not be zero: a cautionary note on the use of inter-rater reliability in evaluating grant peer review”  In _Journal of the Royal Statistical Society: Series A (Statistics in Society)_ 184.3, 2021, pp. 904–919  DOI: 10.1111/rssa.12681
  * [71] Colin Fay, Vincent Guyader, Sébastien Rochette and Cervan Girard  “golem: a framework for robust Shiny applications” R package version 0.4.1, 2023  URL: https://CRAN.R-project.org/package=golem
  * [72] Ian Fellows  “Deducer: a data analysis GUI for R”  In _Journal of Statistical Software_ 49.8, 2012, pp. 1–15  DOI: 10.18637/jss.v049.i08
  * [73] John Fox  “The R commander: a basic-statistics graphical user interface to R”  In _Journal of Statistical Software_ 14.9, 2005, pp. 1–42  DOI: 10.18637/jss.v014.i09
  * [74] Jim Hester et al.  “lintr: a ’linter’ for R code” R package version 3.1.1, 2023  URL: https://github.com/r-lib/lintr
  * [75] Adéla Hladká and Patrícia Martinková  “difNLR: Generalized logistic regression models for DIF and DDF detection”  In _The R Journal_ 12.1, 2020, pp. 300–323  DOI: 10.32614/RJ-2020-014
  * [76] Adéla Hladká, Patrícia Martinková and Marek Brabec  “Parameter estimation in generalised logistic model with application to DIF detection”  In _arXiv_ , 2023  DOI: 10.48550/arXiv.2302.12648
  * [77] Frederic M Lord  “Applications of item response theory to practical testing problems”  Milton Park, Abingdon, Oxon: Routledge, 1980  DOI: 10.4324/9780203056615
  * [78] Jonathon Love  “jmvtools: Tools to build jamovi modules” R package version 2.5.1, 2024  URL: https://github.com/jamovi/jmvtools
  * [79] Jonathon Love et al.  “JASP: Graphical statistical software for common statistical designs”  In _Journal of Statistical Software_ 88.2, 2019, pp. 1–17  DOI: 10.18637/jss.v088.i02
  * [80] D. Magis, S. Beland, F. Tuerlinckx and P. De Boeck  “A general framework and an R package for the detection of dichotomous differential item functioning”  In _Behavior Research Methods_ 42, 2010, pp. 847–862  DOI: 10.3758/brm.42.3.847
  * [81] David Magis and Juan Ramon Barrada  “Computerized adaptive testing with R: Recent updates of the package catR”  In _Journal of Statistical Software, Code Snippets_ 76.1, 2017, pp. 1–19  DOI: 10.18637/jss.v076.c01
  * [82] David Magis and Gilles Raîche  “Random generation of response patterns under computerized adaptive testing with the R package catR”  In _Journal of Statistical Software_ 48.8, 2012, pp. 1–31  DOI: 10.18637/jss.v048.i08
  * [83] David Magis, Duanli Yan and Alina A Von Davier  “Computerized adaptive and multistage testing with R: Using packages catR and mstR”  Springer, 2017  DOI: 10.1007/978-3-319-69218-0
  * [84] Nathan Mantel and William Haenszel  “Statistical aspects of the analysis of data from retrospective studies”  In _Journal of the National Cancer Institute_ 22.4, 1959, pp. 719–748  DOI: 10.1093/jnci/22.4.719
  * [85] Patricia Martinková and Adéla Hladká  “Computational aspects of psychometric methods: With R”  Chapman and Hall/CRC, 2023  DOI: 10.1201/9781003054313
  * [86] Patrícia Martinková, František Bartoš and Marek Brabec  “Assessing inter-rater reliability with heterogeneous variance components models: Flexible approach accounting for contextual variables”  In _Journal of Educational and Behavioral Statistics_ 48.3, 2023, pp. 349–383  DOI: 10.3102/10769986221150517
  * [87] Patrícia Martinková and Adéla Drabinová  “ShinyItemAnalysis for teaching psychometrics and to enforce routine analysis of educational tests”  In _The R Journal_ 10.2, 2018, pp. 503–515  DOI: 10.32614/RJ-2018-074
  * [88] Patrícia Martinková, Adéla Hladká and Eva Potužníková  “Is academic tracking related to gains in learning competence? Using propensity score matching and differential item change functioning analysis for better understanding of tracking implications”  In _Learning and Instruction_ 66 Elsevier, 2020, pp. 101286  DOI: 10.1016/j.learninstruc.2019.101286
  * [89] Patrícia Martinková and Jan Netík  “SIAmodules: Modules for ’ShinyItemAnalysis”’ R package version 0.1.1, 2024  URL: https://cran.r-project.org/package=SIAmodules
  * [90] Edgar C. Merkle, Ellen Fitzsimmons, James Uanhoro and Ben Goodrich  “Efficient Bayesian structural equation modeling in Stan”  In _Journal of Statistical Software_ 100.6, 2021, pp. 1–22  DOI: 10.18637/jss.v100.i06
  * [91] Edgar C. Merkle and Yves Rosseel  “blavaan: Bayesian structural equation models via parameter expansion”  In _Journal of Statistical Software_ 85.4, 2018, pp. 1–30  DOI: 10.18637/jss.v085.i04
  * [92] Jan Netík, Jana Dlouhá and Patrícia Martinková  “EduTestTextAnalysis: Predicting multiple-choice item difficulty from text” R package version 0.1.9, 2024  URL: https://www.ShinyItemAnalysis.org
  * [93] Jan Netík and Patrícia Martinková  “EduTestItemAnalysis: Tools to support psychometric analyses at ’CERMAT”’ R package version 0.1.4, 2024  URL: https://www.ShinyItemAnalysis.org
  * [94] Jan Netík and Patrícia Martinková  “SIAtools: "ShinyItemAnalysis" modules development toolkit” R package version 0.1.1, 2024  URL: https://cran.r-project.org/package=SIAtools
  * [95] R Core Team  “R: a language and environment for statistical computing”, 2024  R Foundation for Statistical Computing  URL: https://www.R-project.org/
  * [96] Nambury S Raju  “The area between two item characteristic curves”  In _Psychometrika_ 53.4 Springer, 1988, pp. 495–502  DOI: 10.1007/bf02294403
  * [97] Nambury S Raju  “Determining the significance of estimated signed and unsigned areas between two item response functions”  In _Applied Psychological Measurement_ 14.2 Sage Publications Sage CA: Thousand Oaks, CA, 1990, pp. 197–207  DOI: 10.1177/014662169001400208
  * [98] Calyampudi Radhakrishna Rao and Sandip Sinharay  “Psychometrics” 26, Handbook of statistics  Amsterdam; Boston: Elsevier, 2007
  * [99] William Revelle  “psych: Procedures for psychological, psychometric, and personality research” R package version 2.4.3  Evanston, Illinois: Northwestern University, 2024  URL: https://CRAN.R-project.org/package=psych
  * [100] Stefan Rödiger, Thomas Friedrichsmeier, Prasenjit Kapat and Meik Michalke  “RKWard: a comprehensive graphical user interface and integrated development environment for statistical analysis with R”  In _Journal of Statistical Software_ 49.9, 2012, pp. 1–34  DOI: 10.18637/jss.v049.i09
  * [101] Yves Rosseel  “lavaan: An R package for structural equation modeling”  In _Journal of Statistical Software_ 48.2, 2012, pp. 1–36  DOI: 10.18637/jss.v048.i02
  * [102] Ravi Selker, Jonathon Love and Damian Dropmann  “jmv: The ’jamovi’ analyses” R package version 2.4.11, 2023  URL: https://CRAN.R-project.org/package=jmv
  * [103] Robin Shealy and William Stout  “A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF”  In _Psychometrika_ 58.2 Springer, 1993, pp. 159–194  DOI: 10.1007/bf02294572
  * [104] Lubomír Štěpánek, Jana Dlouhá and Patrícia Martinková  “Item difficulty prediction using item text features: Comparison of predictive performance across machine-learning algorithms”  In _Mathematics_ 11.4104, 2023  DOI: 10.3390/math11194104
  * [105] Hariharan Swaminathan and H Jane Rogers  “Detecting differential item functioning using logistic regression procedures”  In _Journal of Educational Measurement_ 27.4, 1990, pp. 361–370  DOI: 10.1111/j.1745-3984.1990.tb00754.x
  * [106] The jamovi project  “jamovi” Computer software, version 2.5, 2024  URL: https://www.jamovi.org
  * [107] Alina A Davier, Robert J Mislevy and Jiangang Hao  “Computational psychometrics: New methodologies for a new generation of digital learning and assessment: With examples in R and python”  Springer Nature, 2022
  * [108] Hadley Wickham  “Mastering Shiny: Build interactive apps, reports, and dashboards power by R”  Sebastopol, CA: O’Reilly Media, 2021, pp. 259–286
  * [109] Hadley Wickham and Jenny Bryan  “R Packages: Organize, Test, Document, and Share Your Code”  Beijing Boston Farnham Sebastopol Tokyo: O’Reilly Media, 2023
  * [110] Hadley Wickham, Peter Danenberg, Gábor Csárdi and Manuel Eugster  “roxygen2: In-line documentation for R” R package version 7.3.1, 2024  URL: https://CRAN.R-project.org/package=roxygen2
  * [111] Jan Wijffels and Kohei Watanabe  “word2vec: Distributed representations of words” R package version 0.4.0, 2023  URL: https://CRAN.R-project.org/package=word2vec
  * [112] Kamil Żyła et al.  “rhino: a framework for enterprise Shiny applications” R package version 1.7.0, 2024  URL: https://CRAN.R-project.org/package=rhino