Preferred Name

Mary M. Johnston

ORCID

http://orchid.org/0000-0003-3556-4415

Date of Award

Summer 2016

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Department of Graduate Psychology

Advisor(s)

Dena A. Pastor

Abstract

Noncognitive measures are increasingly being used for accountability purposes in higher education (e.g., O. L. Liu, Frankel, & Roohr, 2014). Because these measures are often collected under low-stakes conditions, there is a concern students do not put forth their best effort when responding, which is problematic given previous research has found noneffortful responding can negatively impact the validity of results (e.g., Barry & Finney, 2009; Meade & Craig, 2012; Swerdzewski, Harmes, & Finney, 2011). Subsequently, there is a need to identify students displaying low effort on low-stakes noncognitive measures. One method, which is based on response time and can discreetly assess student effort at the item level, is the solution behavior (SB) index (Kong, Wise, & Bhola, 2007). A challenging task in using the SB index is the identification of an appropriate time threshold that can meaningfully distinguish responses made with effort (i.e., solution behavior responses) from responses made without effort (i.e., rapid responses). Thus, the purpose of the current study was to examine if the SB index could be used with low-stakes noncognitive measures to distinguish responses – and ultimately students – exhibiting solution behavior from responses made without any effort. In particular, eight different time threshold calculation methods were used to classify responses to a noncognitive measure assessing the construct meaningful life. The resulting time thresholds and SB classification indices were compared and external validity evidence for the resulting SB classification indices was gathered. Results of the study found support for four of the eight threshold calculation methods. In particular, support was found for defining the time thresholds by (a) visually inspecting items’ response time distributions, (b) visually inspecting items’ response time distributions with a known group of rapid responders added to the sample, (c) using a normative threshold that was 30% of the average response time to an item, and (d) using lognormal mixture modeling. Practical implications and limitations of the results are discussed.

Share

COinS
 
 

To view the content in your browser, please download Adobe Reader or, alternately,
you may Download the file to your hard drive.

NOTE: The latest versions of Adobe Reader do not support viewing PDF files within Firefox on Mac OS and if you are using a modern (Intel) Mac, there is no official plugin for viewing PDF files within the browser window.