Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
ORCID
https://orcid.org/0009-0006-8158-9428
Date of Graduation
5-14-2026
Semester of Graduation
Spring
Degree Name
Doctor of Philosophy (PhD)
Department
Department of Graduate Psychology
First Advisor
Christine DeMars
Second Advisor
Sara Finney
Third Advisor
Dena Pastor
Abstract
In low-stakes assessment contexts, test scores are vulnerable to construct-irrelevant variance when examinees vary in their test-taking motivation. To address this concern, researchers have developed both behavioral, time-based indices and self-report measures to identify disengaged examinees; however, prior work consistently shows that these approaches converge modestly but still identify distinct subsets of students. The present study evaluates the performance of a single dichotomous self-report attention item, Use My Data (UMD), as an indicator of disengagement in a low-stakes cognitive assessment context and examines how it compares to an established behavioral index, response time effort (RTE), and a multi-item self-report scale, the Student Opinion Scale-Effort (SOS-Effort). Data were drawn from 3,276 graduating seniors who completed a low-stakes institutional accountability testing session during the 2024-2025 academic year. Four research questions examined (a) agreement among engagement classifications derived from UMD, RTE, and SOS-Effort; (b) changes in score distributions following motivation filtering; (c) associations between engagement indices and student characteristics; and (d) whether student characteristics distinguished cross-classified engagement groups. Results indicated a convergent-but-nonredundant pattern across indices. RTE identified the largest proportion of disengaged examinees and demonstrated the strongest relationship to test performance, producing the largest shifts in score distributions after filtering. SOS-Effort flagged the fewest examinees and showed relatively weak differentiation by academic performance. UMD occupied an intermediate position: it aligned with both RTE and SOS-Effort while also identifying a meaningful subset of students not detected by either index. Discriminant analyses further suggested that UMD reflects an evaluative judgment about data usability influenced by both dispositional and situational factors, rather than a purely behavioral response process. Together, these findings support interpreting UMD as a pragmatic, transparent screening tool that captures a distinct manifestation of disengagement. The results underscore the multifaceted nature of engagement and suggest that combining behavioral and self-report indicators may offer the most defensible approach to motivation filtering in low-stakes assessment programs.
