Putting Students in Their Places: Application of Transitional Diagnostic Classification Model in Higher Education Assessment
Faculty Advisor Name
Dena Pastor
Department
Department of Graduate Psychology
Description
Many programs at James Madison University use Assessment Day to collect student performance data regarding their student learning objectives. Typically, programs use a single test designed to assess multiple objectives. Programs expect to receive usable information on each objective. Typically, a program uses a single test designed to assess multiple student learning objectives. However, objective-level scores are unreliable (untrustworthy) and thus the program receives a single total score.
To address this issue, we applied the transitional diagnostic classification models (TDCM) to an Assessment Day dataset (n = 1710) from the Sociocultural Dimension Assessment version 6 (SDA6; Halonen et al., 2005). Rather than assigning continuous scores for each student, this model allows us to classify students into learning mastery groups (mastery and non-mastery). Under this type of model, a program can obtain proportions of students achieving learning mastery on individual objectives.
The TDCM is a type of statistical model belonging to the family of Diagnostic Classification Models (DCMs). These models classify students into groups based on their test performance. Although these models are used in K-12 education literature, they are not commonly applied in higher education. Perhaps this is due to a combination of the complexity and accessibility of these models as well as a lack of awareness. We were only aware of a single study that used a DCM in higher education assessment (Jurich & Bradshaw, 2014). In their study, Jurich and Bradshaw demonstrated the utility of DCMs by fitting the model to both pretest and posttest data collected from JMU’s Assessment Day. However, in their study, Jurich and Bradshaw assumed that test items relate to their respective objectives in the same way at both pretest and posttest. If this is not the case students will be incorrectly classified into mastery groups. The TDCM is better because it does not make this assumption.
When evaluating pre-post data, the TDCM allows analysts to test whether the relations between items and their objectives are equivalent at both time points. Further, this model predicts group membership at posttest based on group membership at pretest. Using this info, this model can estimate how many students mastered an objective over time. We believe that this information is useful to program stakeholders. Instead of looking at means of student test scores at different time points, stakeholders can see how many students achieved learning mastery at posttest under TDCM. Additionally, with a small number of items we can more accurately assign students to groups than assign individual scores. Thus, programs may shorten their tests while still achieving adequate reliability.
In sum, we applied a new statistical model (TDCM) to Assessment day data. Using this model, program stakeholders can receive trustworthy information on student learning for each individual student learning objectives. Further, stakeholders can examine student learning growth at different time points.
Putting Students in Their Places: Application of Transitional Diagnostic Classification Model in Higher Education Assessment
Many programs at James Madison University use Assessment Day to collect student performance data regarding their student learning objectives. Typically, programs use a single test designed to assess multiple objectives. Programs expect to receive usable information on each objective. Typically, a program uses a single test designed to assess multiple student learning objectives. However, objective-level scores are unreliable (untrustworthy) and thus the program receives a single total score.
To address this issue, we applied the transitional diagnostic classification models (TDCM) to an Assessment Day dataset (n = 1710) from the Sociocultural Dimension Assessment version 6 (SDA6; Halonen et al., 2005). Rather than assigning continuous scores for each student, this model allows us to classify students into learning mastery groups (mastery and non-mastery). Under this type of model, a program can obtain proportions of students achieving learning mastery on individual objectives.
The TDCM is a type of statistical model belonging to the family of Diagnostic Classification Models (DCMs). These models classify students into groups based on their test performance. Although these models are used in K-12 education literature, they are not commonly applied in higher education. Perhaps this is due to a combination of the complexity and accessibility of these models as well as a lack of awareness. We were only aware of a single study that used a DCM in higher education assessment (Jurich & Bradshaw, 2014). In their study, Jurich and Bradshaw demonstrated the utility of DCMs by fitting the model to both pretest and posttest data collected from JMU’s Assessment Day. However, in their study, Jurich and Bradshaw assumed that test items relate to their respective objectives in the same way at both pretest and posttest. If this is not the case students will be incorrectly classified into mastery groups. The TDCM is better because it does not make this assumption.
When evaluating pre-post data, the TDCM allows analysts to test whether the relations between items and their objectives are equivalent at both time points. Further, this model predicts group membership at posttest based on group membership at pretest. Using this info, this model can estimate how many students mastered an objective over time. We believe that this information is useful to program stakeholders. Instead of looking at means of student test scores at different time points, stakeholders can see how many students achieved learning mastery at posttest under TDCM. Additionally, with a small number of items we can more accurately assign students to groups than assign individual scores. Thus, programs may shorten their tests while still achieving adequate reliability.
In sum, we applied a new statistical model (TDCM) to Assessment day data. Using this model, program stakeholders can receive trustworthy information on student learning for each individual student learning objectives. Further, stakeholders can examine student learning growth at different time points.
