Investigating the direction of item parameter bias in the presence of rapid guessing

Authors

• Sarah Alahmadi, James Madison University

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.

ORCID

https://orcid.org/0000-0002-9985-6807

Date of Graduation

5-9-2024

Semester of Graduation

Spring

Degree Name

Doctor of Philosophy (PhD)

Department

Department of Graduate Psychology

First Advisor

Christine E. DeMars

Second Advisor

Brian C. Leventhal

Third Advisor

Brian C. Leventhal

Abstract

Inadequate examinee test-taking effort is especially concerning when the results of low-stakes tests are used to inform high-stakes policy or psychometric decisions. Although abundant is the literature on the negative impact of low test-taking effort (e.g., Wise & DeMars, 2005) and ways to address it (e.g., Alahmadi & DeMars, 2024; Lau et al., 2009), integrating it into assessment practices has long been in a rudimentary stage. The findings presented in this dissertation serve as a clarion call for testing professionals to reinforce efforts that bridge research and practice. The current study investigated how rapid guessing (RG)—a form of non-effortful test-taking behavior—biases the estimated item parameters (i.e., the discrimination and difficulty parameters). Previous studies disagreed on the direction of the bias and the contributing factors. Using simulation techniques to mimic data observed in real-world, low-stakes testing contexts, the current study reconciled the inconsistencies by identifying conditions under which item parameters are likely to be overestimated or underestimated, given several factors. Some factors represent item features (viz., the true values of item parameters and the number of RG responses on items) and others represent examinee features (viz., whether rapid guessers exhibit lower proficiency, the variability in RG behavior among rapid guessers, and the pattern of RG responses throughout the test). The current findings accentuate the consequences of ignoring rapid guessing by revealing implications not only in estimating proficiency scores but also in test operations. The dissertation concludes with a necessary discussion on how the presence of rapid guessing may impair operational procedures such as adaptive testing, equating, and standard setting. Solutions for testing professionals and future directions for researchers are offered.