Exactly! Remember too that these values don't change when you convert to z- or t-scores because this conversion does not change the shape of the distribution.
Faculty Advisor Name
Megan Good
Department
Department of Graduate Psychology
Description
Postsecondary education has been heralded as a catalyst of personal and societal growth. As of 2023, 18.9 million Americans attend a postsecondary institution (Hanson, 2023a). The average cost of college for a student in the United States per year is $36,436 including daily living expenses, supplies, and books (Hanson, 2023b). Consequently, many students borrow to afford their education; the average federal student loan debt in the U.S. is $37,787 as of 2023 (Hanson, 2023b). In addition , the median starting salary after earning a four-year degree is $61,600 (National Center for Education Statistics, 2023). Unsurprisingly, there are questions about the value of a college education. Americans are questioning whether the benefits of college are worth the ever-growing costs (Belkin, 2023; Dickler, 2023; Quintana, 2022). College is expensive, and stakeholders are interested in evidence of higher education’s value. One way to measure return on investment is to consider career outcomes. It is of interest to know if positive career outcomes are – at least in part – a function of college.
There is a need for psychometrically sound measures of career outcomes in higher education. The National Alumni Career Mobility (NACM) survey is in wide use and psychometric evidence is lacking. I evaluated the NACM using Benson’s (1998) three stages of construct validity (i.e., substantive, structural, and external). Regarding substantive evidence, the NACM lacks a theoretical foundation for its five factors of alumni career mobility. In addition, it is unclear why they report a career mobility index (CMI) and why it encompasses three of the five factors of career outcomes. As a result, the substantive stage is insufficient.
Structural evidence is also lacking. Two, one-page psychometric reports summarized evidence of the NACM’s factor structure. While the models showed good fit, there are many questions surrounding the methodology used to generate these reports. Notably, it’s unclear if a five-factor model or a bi-factor model that includes the “CMI” should be used for score reporting. Notably, first-generation and non-first-generation alumni are compared on the NACM subscales without any evidence of measurement invariance.
Regarding the external stage, the CLC examined the NACM subscales in relation to relevant variables (i.e., career advice). However, given the unsound evidence for the substantive and structural stages and the lack of a priori predictions, these relationships cannot warrant external stage evidence. Given that the NACM is widely used and reported on, providing structural stage evidence for score reports is the focus of the current project.
The purpose of the current study was twofold: First, to identify if the bifactor CMI structure fits the NACM data above and beyond the five-factor structure that was initially implemented (dimensionality). Second, to test measurement invariance on the championed model between first-generation and non-first-generation students (measurement invariance). Evidence was provided for an adjusted five-factor model that removes a problematic item from the NACM scale. This model fit the data well and implies that five subscales should be utilized in national reporting: career pathway preparation, career satisfaction, economic mobility, community engagement, and institutional career investment. The CMI should not be reported for the NACM since it lacks both theoretical and structural evidence. Second, evidence of measurement invariance was provided for first-generation and non-first-generation alumni. It was found that the NACM possesses strong (scalar) measurement invariance for these two groups. Thus, it is appropriate to compare scores of first-generation alumni to non-first-generation alumni on the NACM subscales.
Exactly! Remember too that these values don't change when you convert to z- or t-scores because this conversion does not change the shape of the distribution.
Postsecondary education has been heralded as a catalyst of personal and societal growth. As of 2023, 18.9 million Americans attend a postsecondary institution (Hanson, 2023a). The average cost of college for a student in the United States per year is $36,436 including daily living expenses, supplies, and books (Hanson, 2023b). Consequently, many students borrow to afford their education; the average federal student loan debt in the U.S. is $37,787 as of 2023 (Hanson, 2023b). In addition , the median starting salary after earning a four-year degree is $61,600 (National Center for Education Statistics, 2023). Unsurprisingly, there are questions about the value of a college education. Americans are questioning whether the benefits of college are worth the ever-growing costs (Belkin, 2023; Dickler, 2023; Quintana, 2022). College is expensive, and stakeholders are interested in evidence of higher education’s value. One way to measure return on investment is to consider career outcomes. It is of interest to know if positive career outcomes are – at least in part – a function of college.
There is a need for psychometrically sound measures of career outcomes in higher education. The National Alumni Career Mobility (NACM) survey is in wide use and psychometric evidence is lacking. I evaluated the NACM using Benson’s (1998) three stages of construct validity (i.e., substantive, structural, and external). Regarding substantive evidence, the NACM lacks a theoretical foundation for its five factors of alumni career mobility. In addition, it is unclear why they report a career mobility index (CMI) and why it encompasses three of the five factors of career outcomes. As a result, the substantive stage is insufficient.
Structural evidence is also lacking. Two, one-page psychometric reports summarized evidence of the NACM’s factor structure. While the models showed good fit, there are many questions surrounding the methodology used to generate these reports. Notably, it’s unclear if a five-factor model or a bi-factor model that includes the “CMI” should be used for score reporting. Notably, first-generation and non-first-generation alumni are compared on the NACM subscales without any evidence of measurement invariance.
Regarding the external stage, the CLC examined the NACM subscales in relation to relevant variables (i.e., career advice). However, given the unsound evidence for the substantive and structural stages and the lack of a priori predictions, these relationships cannot warrant external stage evidence. Given that the NACM is widely used and reported on, providing structural stage evidence for score reports is the focus of the current project.
The purpose of the current study was twofold: First, to identify if the bifactor CMI structure fits the NACM data above and beyond the five-factor structure that was initially implemented (dimensionality). Second, to test measurement invariance on the championed model between first-generation and non-first-generation students (measurement invariance). Evidence was provided for an adjusted five-factor model that removes a problematic item from the NACM scale. This model fit the data well and implies that five subscales should be utilized in national reporting: career pathway preparation, career satisfaction, economic mobility, community engagement, and institutional career investment. The CMI should not be reported for the NACM since it lacks both theoretical and structural evidence. Second, evidence of measurement invariance was provided for first-generation and non-first-generation alumni. It was found that the NACM possesses strong (scalar) measurement invariance for these two groups. Thus, it is appropriate to compare scores of first-generation alumni to non-first-generation alumni on the NACM subscales.
