[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.20] at ( 62.63, 23.83) -2;

[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.20] at ( 98.43, 23.83) 0;

[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.20] at (134.23, 23.83) 2;

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[draw=drawColor,line width= 0.6pt,line join=round] (193.84, 34.29) –
(193.84, 37.04);

[draw=drawColor,line width= 0.6pt,line join=round] (226.89, 34.29) –
(226.89, 37.04);

[draw=drawColor,line width= 0.6pt,line join=round] (259.93, 34.29) –
(259.93, 37.04);

[draw=drawColor,line width= 0.6pt,line join=round] (292.98, 34.29) –
(292.98, 37.04);

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.20] at (193.84, 23.83) -2.5;

[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.20] at (226.89, 23.83) 0.0;

[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.20] at (259.93, 23.83) 2.5;

[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.20] at (292.98, 23.83) 5.0;

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[draw=drawColor,line width= 0.6pt,line join=round] (309.84, 34.29) –
(309.84, 37.04);

[draw=drawColor,line width= 0.6pt,line join=round] (340.96, 34.29) –
(340.96, 37.04);

[draw=drawColor,line width= 0.6pt,line join=round] (372.08, 34.29) –
(372.08, 37.04);

[draw=drawColor,line width= 0.6pt,line join=round] (403.20, 34.29) –
(403.20, 37.04);

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.20] at (309.84, 23.83) -5.0;

[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.20] at (340.96, 23.83) -2.5;

[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.20] at (372.08, 23.83) 0.0;

[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.20] at (403.20, 23.83) 2.5;

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[text=drawColor,anchor=base east,inner sep=0pt, outer sep=0pt, scale=  1.20] at ( 33.49, 43.13) 7;

[text=drawColor,anchor=base east,inner sep=0pt, outer sep=0pt, scale=  1.20] at ( 33.49, 64.46) 8;

[text=drawColor,anchor=base east,inner sep=0pt, outer sep=0pt, scale=  1.20] at ( 33.49, 85.78) 9;

[text=drawColor,anchor=base east,inner sep=0pt, outer sep=0pt, scale=  1.20] at ( 33.49,107.11) 10;

[text=drawColor,anchor=base east,inner sep=0pt, outer sep=0pt, scale=  1.20] at ( 33.49,128.43) 11;

[text=drawColor,anchor=base east,inner sep=0pt, outer sep=0pt, scale=  1.20] at ( 33.49,149.76) 12;

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[draw=drawColor,line width= 0.6pt,line join=round] ( 35.69, 47.26) –
( 38.44, 47.26);

[draw=drawColor,line width= 0.6pt,line join=round] ( 35.69, 68.59) –
( 38.44, 68.59);

[draw=drawColor,line width= 0.6pt,line join=round] ( 35.69, 89.91) –
( 38.44, 89.91);

[draw=drawColor,line width= 0.6pt,line join=round] ( 35.69,111.24) –
( 38.44,111.24);

[draw=drawColor,line width= 0.6pt,line join=round] ( 35.69,132.57) –
( 38.44,132.57);

[draw=drawColor,line width= 0.6pt,line join=round] ( 35.69,153.89) –
( 38.44,153.89);

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.50] at (233.28,  8.42) $\eta_T$;

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[text=drawColor,rotate= 90.00,anchor=base,inner sep=0pt, outer sep=0pt, scale=  1.50] at ( 15.83, 96.25) Posttest Score;

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[fill=fillColor] (129.46,185.86) rectangle (337.11,211.31);

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[fill=fillColor] (142.46,191.36) rectangle (178.59,205.81);

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[draw=drawColor,line width= 0.4pt,line join=round,line cap=round,fill=fillColor] (160.53,198.58) circle (  3.57);

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[draw=drawColor,line width= 2.3pt,line join=round] (142.46,191.36) – (178.59,205.81);

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[fill=fillColor] (233.28,191.36) rectangle (269.42,205.81);

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[draw=drawColor,line width= 0.4pt,line join=round,line cap=round,fill=fillColor] (251.35,198.58) circle (  3.57);

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[draw=drawColor,line width= 2.3pt,line join=round] (233.28,191.36) – (269.42,205.81);

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[text=drawColor,anchor=base west,inner sep=0pt, outer sep=0pt, scale=  1.20] at (186.09,194.45) Control;

[clip] (  0.00,  0.00) rectangle (433.62,216.81);

[text=drawColor,anchor=base west,inner sep=0pt, outer sep=0pt, scale=  1.20] at (276.92,194.45) Treatment;

One random draw of $\eta_T$ (estimated for the Immediate group, imputed for the Delayed group) or $logit(\hat{p})$ (calculated or imputed) plotted against observed posttest scores. The distance between the two trend-lines at a value of $\eta_T$ is the principal effect.

§ DISCUSSION

Measurement of program implementation and implementation fidelity in an RCT can be complex and nuanced, combining different types of data and different data sources <cit.>e.g.>goodson2014measuring,schoenwald2011toward,carroll2007conceptual,vanacore_ottmar_liu_sales_2023.
The rich log data produced by technology-based interventions such as EdTech demands even greater care in implementation modeling.
This paper evaluated FLPS, a new approach to incorporating these high-quality measurements of implementation into the causal modeling process, allowing researchers to estimate the moderating role of implementation in an intervention's effectiveness.
Given an implementation model, FLPS attempts to use baseline covariates to impute potential implementation $\eta_T$ for subjects assigned to the control condition, and then estimate an interaction between $\eta_T$ and treatment assignment in a causal model.
The Bayesian approach to model fitting, which considers the implementation, imputation, and causal models as interconnected facets of a larger model, ensures that uncertainty is properly propagated.

Or, at least, it should.
It can be difficult to tell if complex models such as FLPS truly estimate what we want them to estimate.
This is especially true in the PS context, where there is cause for concern that even well-specified models may yield severely biased estimates <cit.>.
These concerns motivated our extensive simulation study.

Our simulation study showed that the FLPS worked properly across the simulation conditions with different IRT models as measurement models, although the 2PL model and the GPCM showed low convergence rates. We note that $\hat{R}$s of the nonconvergent estimates from these models ranged between 1.1 and 1.3 and may achieve a higher convergence rate as longer MCMC runs are implemented (see Section 2 in the supplementary materials for additional details).

With regard to the estimation accuracy, all model parameter estimates showed adequate accuracy across the conditions. In general, it was found that increasing sample size leads to more accurate estimation. Increasing the measurement size (e.g., the number of items) improved the recovery of the latent variable and led to minor differences in the item and structural model parameter estimates.
The principal effect, which is the primary concern of principal stratification, showed adequate accuracy across the conditions. Sample size was found to be the most influential factor for estimating the FLPS parameters, followed by the type of measurement model. In terms of coverage rate, all conditions resulted in coverage rates over 0.9, except for one condition: the GPCM with $N$ = 250 and $J$ = 100 condition. The use of the GPCM in a small sample had lower coverage in item intercepts due to the influence of selected priors. In this study, the two polytomous IRT models, GPCM and GRM, had similar overall bias and RMSE, but differed in coverage rates, which requires further research.

We also illustrated the use of FLPS in analyzing data from an RCT comparing an EdTech application that provided immediate feedback to struggling students to one that did not.
There appeared to be little to no effect of immediate feedback on students' state test scores—our FLPS results showed that this null result was probably not due to low dosage (i.e. too little feedback), but likely reflected ineffectiveness for other reasons.
All three measurement models we considered gave strikingly similar estimates, suggesting that, at least in this case, FLPS was robust to minor misspecification of the measurement model.

The results of this study suggest that the use of FLPS benefits causal inference estimation. FLPS models will allow researchers to gain deeper and more nuanced insights into the relationship between the effectiveness of the interventions under study—in particular when big implementation data is available, such as in evaluations of computer-based interventions. Moreover, the FLPS framework will facilitate the thorough study of causal mechanisms in various measurement designs, permitting subsequent study designs to be optimized to reduce measurement error.

The success of the models we considered here suggests even more exciting possibilities.
First, the current study only considered the IRT models, which apply to categorical indicators. There are other types of log data that vary in measurement, such as continuous, longitudinal, and more complex data structures. Future work will extend the model to confirmatory factor analysis or latent growth modeling.
Indeed, many different types of latent variable models are available for the FLPS. The current study only considered the continuous latent variable based on the IRT models. Since discrete latent variables are possible options for the principal effects, we hope that future work will also incorporate mixture models into FLPS such as Latent Class Analysis modeling or Factor mixture modeling.

Furthermore, this study was only conducted in terms of a single latent variable. There are more complicated relations within log-data, which multidimensional structures may underlie. It will be worth exploring model formulations for a variety of multidimensional factor models to be included in the FLPS framework in future research.

This study's main limitation was that it considered only well-specified models. For example, we generated the data based on the Rasch model and fitted the Rasch model for such data. In practice, the correct model specification will be unknown, and researchers might use misspecified models mistakenly or for convenience. The results of the real data application suggest that FLPS may be robust to certain measurement model misspecification. Future research can investigate whether this robustness is a general property, and if so, how widely it extends, as well as whether and how the FLPS models may be robust to structural model misspecification. Future work will also investigate the effectiveness of model checking procedures tailored to FLPS.

Acknowledgements. The research reported here was supported by the Institute of Education Sciences, U.S.
Department of Education, through Grant R305D210036. The opinions expressed are those of the authors
and do not represent views of the Institute or the U.S. Department of Education

, Imbens

Angrist, JD.
, Imbens, GW.
Rubin, DB.

Identification of causal effects using instrumental
variables Identification of causal effects using instrumental
Journal of the American statistical

Baker, FB.

An investigation of the item parameter recovery
characteristics of a Gibbs sampling procedure An investigation of the item
parameter recovery characteristics of a gibbs sampling procedure.
Applied Psychological Measurement222153–169.

Birnbaum, AL.

Some latent trait models and their use in inferring an
examinee's ability Some latent trait models and their use in inferring an
examinee's ability.
Statistical theories of mental test scores.

Carroll, C.
, Patterson, M.
, Wood, S.
, Booth, A.
, Rick, J.
Balain, S.

A conceptual framework for implementation fidelity A
conceptual framework for implementation fidelity.
Implementation science21–9.

Cohen, J.

Statistical Power Analysis for the Behavioral Sciences.
Hillsdale. NJ: Lawrence Erihaum Asso-ciates. Statistical power analysis for
the behavioral sciences. hillsdale. nj: Lawrence erihaum asso-ciates.

, Schafer

Collins, LM.
, Schafer, JL.
Kam, CM.

A comparison of inclusive and restrictive strategies in
modern missing data procedures. A comparison of inclusive and restrictive
strategies in modern missing data procedures.
Psychological methods64330.

Decker-Woodrow, LE.
, Mason, CA.
, Lee, JE.
, Chan, JYC.
, Sales, A.
, Liu, A.
Tu, S.

The Impacts of Three Educational Technologies on
Algebraic Understanding in the Context of COVID-19 The impacts of three
educational technologies on algebraic understanding in the context of
AERA Open923328584231165919.

, Grindal
, Miratrix

Feller, A.
, Grindal, T.
, Miratrix, L.
Page, LC.

Compared to what? Variation in the impacts of early
childhood education by alternative care type Compared to what? variation in
the impacts of early childhood education by alternative care type.
The Annals of Applied Statistics1031245–1285.

Frangakis, CE.
Rubin, DB.

Principal stratification in causal inference Principal
stratification in causal inference.

Fyfe, ER.
, de Leeuw, JR.
, Carvalho, PF.
, Goldstone, RL.
, Sherman, J.
, Admiraal, D.
Motz, BA.

ManyClasses 1: Assessing the Generalizable Effect of
Immediate Feedback Versus Delayed Feedback Across Many College Classes
Manyclasses 1: Assessing the generalizable effect of immediate feedback
versus delayed feedback across many college classes.
Advances in Methods and Practices in Psychological

Gallop, R.
, Small, DS.
, Lin, JY.
, Elliott, MR.
, Joffe, M.
Ten Have, TR.

Mediation analysis with principal stratification
Mediation analysis with principal stratification.
Statistics in medicine2871108–1130.

Gelman, A.
, Carlin, JB.
, Stern, HS.
, Dunson, DB.
, Vehtari, A.
Rubin, DB.

Bayesian data analysis Bayesian data analysis ( 2).
CRC press Boca Raton, FL.

Gelman, A.
Rubin, DB.

Inference from iterative simulation using multiple
sequences Inference from iterative simulation using multiple
Statistical science457–472.

, Price

Goodson, B.
, Price, C.
Darrow, C.

Measuring Fidelity: The present and future Measuring
fidelity: The present and future.
B. Myers  WC. Brand (), Implementation Fidelity in Education Research Implementation fidelity in
education research ( 190–207).

, McCaffery

Griffin, BA.
, McCaffery, DF.
Morral, AR.

An application of principal stratification to control
for institutionalization at follow-up in studies of substance abuse treatment
programs An application of principal stratification to control for
institutionalization at follow-up in studies of substance abuse treatment
The annals of applied statistics231034.

Heffernan, NT.
Heffernan, CL.

The ASSISTments ecosystem: Building a platform that
brings scientists and teachers together for minimally invasive research on
human learning and teaching The assistments ecosystem: Building a platform
that brings scientists and teachers together for minimally invasive research
on human learning and teaching.
International Journal of Artificial Intelligence in

Hill, HC.
Erickson, A.

Using implementation fidelity to aid in interpreting
program impacts: A brief review Using implementation fidelity to aid in
interpreting program impacts: A brief review.
Educational Researcher489590–598.

, Feller
, Greif
, Miratrix

Ho, N.
, Feller, A.
, Greif, E.
, Miratrix, L.
Pillai, N.

Weak Separation in Mixture Models and Implications for
Principal Stratification Weak separation in mixture models and implications
for principal stratification.
International Conference on Artificial Intelligence and
Statistics International conference on artificial intelligence and
statistics ( 5416–5458).

Hoffman, MD.
Gelman, A.

The No-U-Turn sampler: adaptively setting path lengths
in Hamiltonian Monte Carlo. The no-u-turn sampler: adaptively setting path
lengths in hamiltonian monte carlo.
J. Mach. Learn. Res.1511593–1623.

Koedinger, KR.
, Baker, RS.
, Cunningham, K.
, Skogsholm, A.
, Leber, B.
Stamper, J.

A data repository for the EDM community: The PSLC
DataShop A data repository for the edm community: The pslc
Handbook of educational data mining4343–56.

, Zhu

Li, S.
, Zhu, Y.
Ellis, R.

The effects of the timing of corrective feedback on the
acquisition of a new linguistic structure The effects of the timing of
corrective feedback on the acquisition of a new linguistic structure.
The Modern Language Journal1001276–295.

Masters, GN.

Partial credit model Partial credit model.
WJ. van der Linden (), Handbook of Item
Response Theory, Volume One Handbook of item response theory, volume one  ( 137–154).
Chapman and Hall/CRC.

, Li

Mattei, A.
, Li, F.
Mealli, F.

Exploiting multiple outcomes in Bayesian principal
stratification analysis with application to the evaluation of a job training
program Exploiting multiple outcomes in bayesian principal stratification
analysis with application to the evaluation of a job training
The Annals of Applied Statistics742336–2360.

Mealli, F.
Mattei, A.

A refreshing account of principal stratification A
refreshing account of principal stratification.
The international journal of biostatistics81.

Muraki, E.

A Generalized Partial Credit Model A generalized
partial credit model.
WJ. van der Linden  RK. Hambleton (),
Handbook of Modern Item Response Theory Handbook of modern
item response theory ( 153–164).
New York, NYSpringer New York.

Neyman, J.

Sur les applications de la théorie des
probabilités aux experiences agricoles: Essai des principes Sur les
applications de la théorie des probabilités aux experiences
agricoles: Essai des principes.
Roczniki Nauk Rolniczych101–51.

Olson, MB.
, McCreedy, EM.
, Baier, RR.
, Shield, RR.
, Zediker, EE.
, Uth, R.
Rudolph, JL.

Measuring implementation fidelity in a
cluster-randomized pragmatic trial: development and use of a quantitative
multi-component approach Measuring implementation fidelity in a
cluster-randomized pragmatic trial: development and use of a quantitative
multi-component approach.

Ostrow, KS.
, Selent, D.
, Wang, Y.
, Van Inwegen, EG.
, Heffernan, NT.
Williams, JJ.

The assessment of learning infrastructure (ALI): the
theory, practice, and scalability of automated assessment The assessment of
learning infrastructure (ali): the theory, practice, and scalability of
automated assessment.
Proceedings of the Sixth International Conference on
Learning Analytics & Knowledge Proceedings of the sixth international
conference on learning analytics & knowledge ( 279–288).

, Feller
, Grindal
, Miratrix

Page, LC.
, Feller, A.
, Grindal, T.
, Miratrix, L.
Somers, MA.

Principal stratification: A tool for understanding
variation in program effects across endogenous subgroups Principal
stratification: A tool for understanding variation in program effects across
endogenous subgroups.
American Journal of Evaluation364514–531.

R Core Team

R Core Team

R Core Team.

R: A Language and Environment for Statistical Computing
R: A language and environment for statistical computing  [].
Vienna, Austria.

Rasch, G.

Studies in mathematical psychology: I. Probabilistic
models for some intelligence and attainment tests. Studies in mathematical
psychology: I. probabilistic models for some intelligence and attainment

Rasch, G.

Probabilistic models for some intelligence and attainment
tests. Probabilistic models for some intelligence and attainment tests.

Rubin, DB.

Bayesian inference for causal effects: The role of
randomization Bayesian inference for causal effects: The role of
The Annals of statistics34–58.

Sales, AC.
Pane, JF.

The role of mastery learning in an intelligent tutoring
system: Principal stratification on a latent variable The role of mastery
learning in an intelligent tutoring system: Principal stratification on a
latent variable.
The Annals of Applied Statistics131420–443.

Sales, AC.
Pane, JF.

Student log-data from a randomized evaluation of
educational technology: A causal case study Student log-data from a
randomized evaluation of educational technology: A causal case study.
Journal of Research on Educational

Samejima, F.

Estimation of latent ability using a response pattern of
graded scores. Estimation of latent ability using a response pattern of
graded scores.
Psychometrika monograph supplement344100.

Schoenwald, SK.
, Garland, AF.
, Chapman, JE.
, Frazier, SL.
, Sheidow, AJ.
Southam-Gerow, MA.

Toward the effective and efficient measurement of
implementation fidelity Toward the effective and efficient measurement of
implementation fidelity.
Administration and Policy in Mental Health and Mental
Health Services Research3832–43.

Stan Development Team

Stan Development Team

Stan Development Team.

RStan: the R interface to Stan. RStan: the R
interface to Stan.
R package version 2.14.1

Stan Development Team

Stan Development Team

Stan Development Team.

RStan: the R interface to Stan. RStan: the R
interface to Stan.
R package version 2.17.3

Stekhoven, DJ.
Bühlmann, P.

MissForest—non-parametric missing value imputation for
mixed-type data Missforest—non-parametric missing value imputation for
mixed-type data.

Stocking, GW.

Bones, Bodies amd Behavior: Essays in Behavioral Anthropology
Bones, bodies amd behavior: Essays in behavioral anthropology ( 5).
Univ of Wisconsin Press.

Swaminathan, H.
Gifford, JA.

Bayesian estimation in the three-parameter logistic
model Bayesian estimation in the three-parameter logistic model.

, Ottmar
, Liu

Vanacore, K.
, Ottmar, E.
, Liu, A.
Sales, AC.

Remote Monitoring of Implementation Fidelity Using Log-File
Data from Multiple Online Learning Platforms. Remote monitoring of
implementation fidelity using log-file data from multiple online learning

van der Linden

van der Linden

van der Linden, WJ.
Hambleton, RK.

Handbook of modern item response theory Handbook of modern
item response theory.
Springer Science & Business Media.

, Muraki
, Mislevy

Zimowski, MF.
, Muraki, E.
, Mislevy, RJ.
Bock, R.

BILOG-MG [computer software] Bilog-mg [computer
Lincolnwood, IL: Scientific Software