Preferred Name
Nikole Gregg
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
Date of Graduation
Spring 2018
Document Type
Thesis
Degree Name
Master of Arts (MA)
Department
Department of Graduate Psychology
Advisor(s)
John Hathcoat
Abstract
Written communication is a skill necessary for not only the success of undergraduate students, but for post-graduates in the workplace. Furthermore, according to employers the writing skills of post-graduates tend to be below expectations. Therefore, the assessment of such skills within higher education is in high demand. Written communication assessments tend to be administered in one of two conditions: 1) course embedded and 2) a low-stakes, non-embedded condition. The current study investigated possible construct-irrelevant variance in writing assessment scores by using data from a mid-sized public university in the Mid-Atlantic region of the United States. Specifically, 157 student products were scored using the Association of American Colleges and Universities’ Written Communication rubric by Multi-State Collaborative trained raters. A final sample size of 57 student products were in the non-embedded assessment condition and 107 student products were in the embedded assessment condition. Differential item functioning analyses were conducted using a Rasch Rating Scale model and an Ordinal Regression wherein Verbal SAT was used an external criterion of ability. Said differently, this study investigated whether students of the same proficiency had different probabilities of receiving particular written communication scores. After controlling for motivation, the results provide evidence of possible differential item functioning for Content Development as well as Genre and Disciplinary Conventions. Students of the same ability tend to obtain higher written communication scores in the non-embedded assessment condition. These results raise concerns about the presence of construct-irrelevant variance aside from motivation. Future research should investigate faculty feedback, allotted time, and task structure as possible sources of construct-irrelevant variance when using low-stakes, non-embedded assessments of written communication.
Recommended Citation
Gregg, Nikole, "Beyond motivation: Differences in score meaning between assessment conditions" (2018). Masters Theses, 2010-2019. 565.
https://commons.lib.jmu.edu/master201019/565
