Large-Scale Assessment During a Pandemic: Results from James Madison University’s Assessment Day
Faculty Advisor Name
Christine DeMars
Department
Department of Graduate Psychology
Description
JMU has been collecting longitudinal data assessing learning outcomes for more than 30 years. Students are assessed twice, initially as incoming first-year students and later after completing 45-70 credit hours. In the past, proctors played an important role on Assessment Days as they ensured that tests were completed properly, noises were minimized, and students were motivated and aware of the importance of the Assessment Day. However, several modifications were necessary for the 2020-2021 academic year (Pastor & Love, 2020). Assessment was conducted remotely due to the COVID-19 pandemic. Incoming students completed the assessments during a three-week window via links they received by email. The format of testing changed from proctored paper-and-pencil assessment to unproctored computer-based assessment.
Conducting Assessment Day online in Fall of 2020 raised validity concerns regarding such low-stakes (i.e., inconsequential to students), unproctored internet test (UIT) administrations. When tests do not bear any direct consequences, and are unproctored, students may expend less effort, which could, in turn, lower their performance. Empirical evidence is mixed on whether UITs affect assessment results. One study that compared examinee performance in proctored versus unproctored online settings found no significant differences between the two (Hollister & Berenson, 2009). Another study reported higher performance in web-based, remote cognitive tests when they were unproctored as opposed to proctored (Karim, Kaminsky, & Behrend, 2014). Conversely, another study examining performance differences on low-stakes online-proctored tests versus online-unproctored tests showed no significant differences in terms of test-taking behavior or test performance (Rios & Liu, 2017).
We explored differences between assessment results in Fall 2020 and previous years (specifically, 2016, 2017, 2018, and 2019). Our research questions were:
1) Did incoming students score differently in Fall 2020 than students in Fall 2016-Fall 2019?
2) If there are differences, are the differences similar across different tests?
3) Did students report different amounts of test-taking effort in Fall 2020 than previous years?
4) Was effort or time spent testing related to test performance?
Our results indicated that low-stakes remote assessment impacted performance on the more cognitively demanding (i.e., longer and more difficult) scientific reasoning assessment but did not significantly impact performance on the less arduous assessments (global issues and history). The 2020 cohort exhibited significantly lower scores on the scientific reasoning test than their counterparts in previous years. Those students also showed a different distribution of effort on the scientific reasoning test than students in previous cohorts—a subgroup of students seemed to have lower effort, producing a bimodal distribution. Additionally, the time students spent testing predicted performance better for the scientific reasoning test in the 2020 cohort compared to the other tests. These results suggest that the amount of cognitive resources required by the test may impact students’ performance in remote unproctored assessments.
Large-Scale Assessment During a Pandemic: Results from James Madison University’s Assessment Day
JMU has been collecting longitudinal data assessing learning outcomes for more than 30 years. Students are assessed twice, initially as incoming first-year students and later after completing 45-70 credit hours. In the past, proctors played an important role on Assessment Days as they ensured that tests were completed properly, noises were minimized, and students were motivated and aware of the importance of the Assessment Day. However, several modifications were necessary for the 2020-2021 academic year (Pastor & Love, 2020). Assessment was conducted remotely due to the COVID-19 pandemic. Incoming students completed the assessments during a three-week window via links they received by email. The format of testing changed from proctored paper-and-pencil assessment to unproctored computer-based assessment.
Conducting Assessment Day online in Fall of 2020 raised validity concerns regarding such low-stakes (i.e., inconsequential to students), unproctored internet test (UIT) administrations. When tests do not bear any direct consequences, and are unproctored, students may expend less effort, which could, in turn, lower their performance. Empirical evidence is mixed on whether UITs affect assessment results. One study that compared examinee performance in proctored versus unproctored online settings found no significant differences between the two (Hollister & Berenson, 2009). Another study reported higher performance in web-based, remote cognitive tests when they were unproctored as opposed to proctored (Karim, Kaminsky, & Behrend, 2014). Conversely, another study examining performance differences on low-stakes online-proctored tests versus online-unproctored tests showed no significant differences in terms of test-taking behavior or test performance (Rios & Liu, 2017).
We explored differences between assessment results in Fall 2020 and previous years (specifically, 2016, 2017, 2018, and 2019). Our research questions were:
1) Did incoming students score differently in Fall 2020 than students in Fall 2016-Fall 2019?
2) If there are differences, are the differences similar across different tests?
3) Did students report different amounts of test-taking effort in Fall 2020 than previous years?
4) Was effort or time spent testing related to test performance?
Our results indicated that low-stakes remote assessment impacted performance on the more cognitively demanding (i.e., longer and more difficult) scientific reasoning assessment but did not significantly impact performance on the less arduous assessments (global issues and history). The 2020 cohort exhibited significantly lower scores on the scientific reasoning test than their counterparts in previous years. Those students also showed a different distribution of effort on the scientific reasoning test than students in previous cohorts—a subgroup of students seemed to have lower effort, producing a bimodal distribution. Additionally, the time students spent testing predicted performance better for the scientific reasoning test in the 2020 cohort compared to the other tests. These results suggest that the amount of cognitive resources required by the test may impact students’ performance in remote unproctored assessments.
