Does the Student Opinion Scale's Effort Subscale Function Differently Across Black and White Students? A Measurement Invariance Study
Faculty Advisor Name
Dena Pastor
Department
Department of Graduate Psychology
Description
As a measure of program effectiveness, higher education institutions may rely on low-stakes testing data from students. One of the largest challenges that comes with using data in low-stakes testing is validating results. Due to tests being low-stakes (i.e., little to no consequences for low performance), students are typically less likely to put forth their best effort. This introduces construct-irrelevant variance in the form of attenuated scores. Since effort is a moderator of test performance, measuring learning outcomes attainment means institutions should also measure the effort put forth by students taking the test(s). More importantly, recent calls for disaggregating data by race means that comparisons of performance scores across race need to be accurately interpreted in ways that don’t reinforce negative stereotypes about racially marginalized students. Therefore, in order to make comparisons of performance across racial groups, we must also make accurate comparisons of exerted effort across racial groups. To make accurate comparisons of exerted effort across racial groups, we must be sure that the measure used to capture effort operates similarly across all racial groups. The process of measurement invariance is used to ensure that a measure operates the same across any number of groups.
This study has two components using a data pool of 5-years’ worth of Black and White students who participated in a low-stakes, university testing session. After engaging in a series of assessments, participants were asked to report their level of exerted effort across the tests using a 5-item effort scale on the Student Opinion Scale (SOS). First, alternative confirmatory factor analysis (CFA) models are fit to the Black and white students’ data separately to find the best-fitting model for each group. Then measurement invariance testing occurred to assess the level which Black and White students conceptualize and respond to items on the Student Opinion Scale’s effort subscale. Results note that the same CFA model fit both Black and White students. When tested for measurement invariance, invariance holds at the strict invariance level, meaning effort can be compared between Black and White students at the latent level. It was found that White students had a higher average latent effort score, which the difference was not statistically but practically significant.
With validity evidence provided that the SOS effort subscale operates similarly for Black and White students, there are implications in both research in race and effort, as well as measuring effort in low-stakes testing.
Does the Student Opinion Scale's Effort Subscale Function Differently Across Black and White Students? A Measurement Invariance Study
As a measure of program effectiveness, higher education institutions may rely on low-stakes testing data from students. One of the largest challenges that comes with using data in low-stakes testing is validating results. Due to tests being low-stakes (i.e., little to no consequences for low performance), students are typically less likely to put forth their best effort. This introduces construct-irrelevant variance in the form of attenuated scores. Since effort is a moderator of test performance, measuring learning outcomes attainment means institutions should also measure the effort put forth by students taking the test(s). More importantly, recent calls for disaggregating data by race means that comparisons of performance scores across race need to be accurately interpreted in ways that don’t reinforce negative stereotypes about racially marginalized students. Therefore, in order to make comparisons of performance across racial groups, we must also make accurate comparisons of exerted effort across racial groups. To make accurate comparisons of exerted effort across racial groups, we must be sure that the measure used to capture effort operates similarly across all racial groups. The process of measurement invariance is used to ensure that a measure operates the same across any number of groups.
This study has two components using a data pool of 5-years’ worth of Black and White students who participated in a low-stakes, university testing session. After engaging in a series of assessments, participants were asked to report their level of exerted effort across the tests using a 5-item effort scale on the Student Opinion Scale (SOS). First, alternative confirmatory factor analysis (CFA) models are fit to the Black and white students’ data separately to find the best-fitting model for each group. Then measurement invariance testing occurred to assess the level which Black and White students conceptualize and respond to items on the Student Opinion Scale’s effort subscale. Results note that the same CFA model fit both Black and White students. When tested for measurement invariance, invariance holds at the strict invariance level, meaning effort can be compared between Black and White students at the latent level. It was found that White students had a higher average latent effort score, which the difference was not statistically but practically significant.
With validity evidence provided that the SOS effort subscale operates similarly for Black and White students, there are implications in both research in race and effort, as well as measuring effort in low-stakes testing.
