You only live up to the standards you set: An evaluation of different approaches to standard setting
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
Date of Graduation
Spring 2017
Document Type
Dissertation
Degree Name
Doctor of Philosophy (PhD)
Department
Department of Graduate Psychology
Advisor(s)
Dena A. Pastor
Keston H. Fulcher
Allison J. Boykin
Abstract
Interpretation of performance in reference to a standard can provide nuanced, finely-tuned information regarding examinee abilities beyond that of just a total score. However, there is a multitude of ways to set performance standards yet little guidance regarding which method operates best and under what circumstances. Traditional methods are the most common approach adopted in practice and heavily involve subject matter experts (SMEs). Two other approaches have been suggested in the literature as alternative ways to set performance standards, although they have yet to be implemented in practice. Data-driven approaches do not involve SMEs but rather rely solely upon statistical techniques to classify examinees into groups. Integrated approaches are a newer standard setting method that combines judgments provided by SMEs with statistical techniques to inform the creation of performance standards. The primary purpose of this dissertation was to describe and illustrate the traditional, data-driven, and integrated approaches used to establish performance standards on tests. A traditional standard setting was conducted using the modified Angoff procedure. Latent class analysis (LCA)—a data-driven classification technique—was performed in which model parameters were first freely estimated to assess the fit of various general LCA models and later constrained to create ordered groups for various ordinal LCA models. The traditional and data-driven standard setting methods were combined to form an “integrated” approach. SMEs’ ratings of expected examinee performance (derived from the modified Angoff standard setting) were used as item difficulty constraints in an integrated LCA model, the Angoff LCA. The results were used to compare examinee classifications from all three approaches and model-data fit amongst the statistically-oriented methods. Although classifications were planned for comparison across all three approaches, issues were encountered with the Angoff LCA. Therefore, the comparisons of primary interest were between the modified Angoff and championed LCA model. The results did not offer a clear-cut decision about which approach to champion. Ultimately, the modified Angoff was selected as the most appropriate standard setting approach for the test administered. Important considerations are offered for researchers who wish to use data-driven models to set standards and ideas are proposed for future research.
Recommended Citation
Strickman, Scott N., "You only live up to the standards you set: An evaluation of different approaches to standard setting" (2017). Dissertations, 2014-2019. 151.
https://commons.lib.jmu.edu/diss201019/151
Included in
Educational Assessment, Evaluation, and Research Commons, Educational Methods Commons, Educational Psychology Commons, Higher Education Commons, Quantitative Psychology Commons, Social Statistics Commons
