What do We Know about Context: An Integrated Analysis of Context Characteristics of Science Assessment Items

Abstract

Prior studies have widely documented the impact of characteristics of contextualized items on students’ test performance. However, the role of individual context characteristics on assessment results is not fully understood. The purpose of this dissertation is twofold. The primary goal is to provide a holistic view on the influence of contextualized items on science assessment results by investigating the effects of six item characteristics in three studies. The second goal is to explore different methodological approaches to understanding effects of item characteristics and advance the psychometric analysis of contextualized items. To this end, three methodological approaches are used: item response theory, cognitive diagnostic modeling, and machine learning. Specifically, the first study addresses a common myth about contextualized items—richer contexts are always better—by experimenting how physics items with increasing levels of richness impact students’ performance. The richness of item contexts was operationalized by level of context abstractness and inclusion of illustrations. The Rasch model in combination with a hierarchical generalized linear model offers a comprehensive interpretation of the influence of context richness, such as the negative relationship between using contextualized-illustrated items and students’ performance. The second study applies cognitive diagnostic models (CDM) to examine the role of context familiarity on students’ mastery pattern of required physics concepts and two item parameters estimated from CDM. Results show familiarity with item context may impact students’ mastery of certain physics concepts and was negatively related to the guessing parameter estimated from CDM. The third study explores the effects of three item characteristics on item difficulty based on NAEP released science items and illustrates the benefits of using cross-validation for model comparison and feature selection. The three item characteristics are cognitive demands due to the assessed topics and science practices, item format (e.g., multiple choice), and linguistic complexity (e.g., average age-of-acquisition for all words in an item). Results confirm the significant influence of item format and cognitive demands on item difficulty. Experiments were conducted to explore potential reasons why linguistic complexity was not uniquely predictive of item difficulty. Practical implications on test development are provided to serve the larger research community.