Process Matters! Novel Approaches to Using Process Data for Psychometric Research

Abstract

Logs of keystrokes, clicks, eye tracking, mouse movement, action sequences, and time stamps – known as “process data” – have the potential to provide new insights into how people are thinking about item tasks and ideas as they respond to them. The purpose of this dissertation is to extend the prior research on novel analytic approaches – namely network and latent class modeling – for incorporating action sequence process data in psychometric research, this time within an educational gaming environment. Specifically, I used time-stamped action records for selected items of an online elementary level math game to demonstrate four methods: 1) computing descriptive network statistics on each student’s item action sequence for predicting student game performance and engagement for a sample of N = 20 fourth grade students that were assessed on a common set of four math modeling items; 2) using inferential network analyses (exponential random graph models) on each student’s item actions for predicting student game performance and engagement (same set of 20 students and items); 3) conducting hidden Markov models (HMMs) on latent state action-to-action transitions for N = 500 students on the first (entry item) of the math modeling game, and comparing transition patterns by student engagement/performance levels; and 4) using latent class analysis (LCA) to group and predict student outcomes (same set of students and entry item). Consistent with previous research, results showed that each approach revealed different insights into item properties and how those properties connect with student performance and engagement outcomes. This said, the unique game environment of this assessment dictated that item properties would be different at different phases of the game, such that the connections between network or latent factor properties would differ with student outcomes. Further, because real data was used, some results may have limited generalizability. As such, future research directions should include evaluating these analytic approaches using simulated data with varied sample sizes, numbers of action sequences, and action sequence sparseness levels. Further, it would be helpful to qualitatively investigate best practices in connecting these types of model results with actionable feedback for assessment developers. Despite these limitations, this dissertation was able to successfully demonstrate the potential usefulness of applying network and latent class analysis approaches in analyzing item action sequence data to better understand item and student outcomes within an online math modeling game environment.