Considerations and Recommendations for Rapid Guessing with Examinee Motivation Filtering in DCMs
Faculty Advisor Name
Yu Bao
Department
Department of Graduate Psychology
Description
This simulation study examines rapid guessing behavior effects on Diagnostic Classification Models (DCMs) within educational assessment contexts. DCMs are a class of measurement models to assess a students’ proficiency levels across multiple skills, providing valuable insights for instructional interventions and remediation efforts by educators. However, the accurate interpretation and utilization of these classifications are threatened by differential student motivation, which can manifest as rapid guessing behavior.
Rapid guessing occurs when students hastily respond to assessment items without genuine effort. If rapid guessing is unaccounted for students’ engagement is conflated with their proficiency levels. This behavior violates an underlying assumption of DCMs that students provide effortful responses, leading to misleading classifications of proficiency levels. The implications of undressing rapid guessing behaviors have shown deleterious effects on item estimates (Rios & Soland, 2021), person estimates (Rios, 2022), and aggregate scores (Rios et al., 2022).
A two-step procedure is followed to account for rapid guessing behaviors, it is 1) identifying responses that are rapid guesses and 2) treating responses that are identified as rapid guesses. A common method for identifying a rapid guess is using a normative threshold approach. In which for a given item, responses < 10% of the mean response time are considered as rapid guesses. For example, if a mean response time for an item is 45 seconds, responses faster than 4.5 seconds are considered rapid guesses. This approach is computationally efficient and has shown effectiveness in previous literature.
Once rapid guesses are identified, practitioners can filter the data at the examinee or the response level. Examinee filtering consists of list wise deleting examinees that rapidly respond to too many items. Response level filtering consists of treating rapid guess responses as missing or incorrect. These methodologies have been examined in IRT literature; however, they have not been expanded to DCM literature where tests are multidimensional with complex items. Thus, this study proposes expanding examinee motivation filtering to a multidimensional context (multiRTE) and proposes an extension of the effort-moderated IRT model to a DCM context that treats rapid guessing as missing.
The simulation study will examine different constructed DCM tests to determine the effect of rapid guessing. Furthermore, it will examine the efficacy of traditional and extensions of examinee and response level filtering. The simulation study manipulated 4 factors: test length, number of attributes, Q-matrix design, and motivation filtering method. The performance of the motivation filtering methods on different test designs will be evaluated using item bias and RMSE. Furthermore, ability parameter accuracy for individual attributes and profiles are examined.
Considerations and Recommendations for Rapid Guessing with Examinee Motivation Filtering in DCMs
This simulation study examines rapid guessing behavior effects on Diagnostic Classification Models (DCMs) within educational assessment contexts. DCMs are a class of measurement models to assess a students’ proficiency levels across multiple skills, providing valuable insights for instructional interventions and remediation efforts by educators. However, the accurate interpretation and utilization of these classifications are threatened by differential student motivation, which can manifest as rapid guessing behavior.
Rapid guessing occurs when students hastily respond to assessment items without genuine effort. If rapid guessing is unaccounted for students’ engagement is conflated with their proficiency levels. This behavior violates an underlying assumption of DCMs that students provide effortful responses, leading to misleading classifications of proficiency levels. The implications of undressing rapid guessing behaviors have shown deleterious effects on item estimates (Rios & Soland, 2021), person estimates (Rios, 2022), and aggregate scores (Rios et al., 2022).
A two-step procedure is followed to account for rapid guessing behaviors, it is 1) identifying responses that are rapid guesses and 2) treating responses that are identified as rapid guesses. A common method for identifying a rapid guess is using a normative threshold approach. In which for a given item, responses < 10% of the mean response time are considered as rapid guesses. For example, if a mean response time for an item is 45 seconds, responses faster than 4.5 seconds are considered rapid guesses. This approach is computationally efficient and has shown effectiveness in previous literature.
Once rapid guesses are identified, practitioners can filter the data at the examinee or the response level. Examinee filtering consists of list wise deleting examinees that rapidly respond to too many items. Response level filtering consists of treating rapid guess responses as missing or incorrect. These methodologies have been examined in IRT literature; however, they have not been expanded to DCM literature where tests are multidimensional with complex items. Thus, this study proposes expanding examinee motivation filtering to a multidimensional context (multiRTE) and proposes an extension of the effort-moderated IRT model to a DCM context that treats rapid guessing as missing.
The simulation study will examine different constructed DCM tests to determine the effect of rapid guessing. Furthermore, it will examine the efficacy of traditional and extensions of examinee and response level filtering. The simulation study manipulated 4 factors: test length, number of attributes, Q-matrix design, and motivation filtering method. The performance of the motivation filtering methods on different test designs will be evaluated using item bias and RMSE. Furthermore, ability parameter accuracy for individual attributes and profiles are examined.
