Preferred Name
Mary M. Johnston
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
ORCID
http://orchid.org/0000-0003-3556-4415
Date of Graduation
Summer 2016
Document Type
Dissertation
Degree Name
Doctor of Philosophy (PhD)
Department
Department of Graduate Psychology
Advisor(s)
Dena A. Pastor
Abstract
Noncognitive measures are increasingly being used for accountability purposes in higher education (e.g., O. L. Liu, Frankel, & Roohr, 2014). Because these measures are often collected under low-stakes conditions, there is a concern students do not put forth their best effort when responding, which is problematic given previous research has found noneffortful responding can negatively impact the validity of results (e.g., Barry & Finney, 2009; Meade & Craig, 2012; Swerdzewski, Harmes, & Finney, 2011). Subsequently, there is a need to identify students displaying low effort on low-stakes noncognitive measures. One method, which is based on response time and can discreetly assess student effort at the item level, is the solution behavior (SB) index (Kong, Wise, & Bhola, 2007). A challenging task in using the SB index is the identification of an appropriate time threshold that can meaningfully distinguish responses made with effort (i.e., solution behavior responses) from responses made without effort (i.e., rapid responses). Thus, the purpose of the current study was to examine if the SB index could be used with low-stakes noncognitive measures to distinguish responses – and ultimately students – exhibiting solution behavior from responses made without any effort. In particular, eight different time threshold calculation methods were used to classify responses to a noncognitive measure assessing the construct meaningful life. The resulting time thresholds and SB classification indices were compared and external validity evidence for the resulting SB classification indices was gathered. Results of the study found support for four of the eight threshold calculation methods. In particular, support was found for defining the time thresholds by (a) visually inspecting items’ response time distributions, (b) visually inspecting items’ response time distributions with a known group of rapid responders added to the sample, (c) using a normative threshold that was 30% of the average response time to an item, and (d) using lognormal mixture modeling. Practical implications and limitations of the results are discussed.
Recommended Citation
Johnston, Mary M., "Applying solution behavior thresholds to a noncognitive measure to identify rapid responders: An empirical investigation" (2016). Dissertations, 2014-2019. 124.
https://commons.lib.jmu.edu/diss201019/124
