On the Uses of Process Data in Psychometric Research: Response Process Validity, Theory-building, and Operational Research

Abstract

Digitally-based assessments create opportunities for collecting moment to moment information about how students are responding to assessment items. This information, called log or process data, has long been regarded as a vast and valuable source of data about student performance. Despite repeated assurances of its vastness and value, process data research has not yet lived up to this promise. Interesting and essential work has been done to explore methods of analyzing process data, but less attention has been paid to how process data, and results from analyses of it, should be used. This dissertation addresses this gap by providing three examples of how process data can be used: to understand how non-construct related item features influence student response processes, to describe domain performance in an undertheorized domain, and to inform operational decisions about digital-only components of assessment items. Special attention will be paid to justifications for using process data in each example, as way of establishing preliminary guidelines to evaluating the use of process data for research purposes.