Examining the GLAT as a Measure of AI Literacy

Authors

• Hunter R. Castleton, James Madison UniversityFollow

Preferred Name

Hunter Castleton

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License

Date of Graduation

5-16-2026

Semester of Graduation

Spring

Degree Name

Master of Arts (MA)

Department

Department of Graduate Psychology

First Advisor

John Hathcoat

Second Advisor

Dena Pastor

Third Advisor

Yu Bao

Abstract

The current study examined the Generative AI Literacy Assessment Test (GLAT) as a promising test of AI literacy due to its content, which is grounded in relevant AI literacy theory, and its design as a cognitive measure of AI literacy. To examine the GLAT, data were collected from a sample of 543 graduating seniors at James Madison University (JMU) who took the GLAT as part of a longer series of tests. Their responses were first analyzed to assess the test’s item characteristics and internal structure. Following this initial analysis, a final item set was established to ensure that low-quality items were excluded from the data analysis. Final scores were then calculated using the final item set, which were used to assess the GLAT’s relationships with other variables. The study found the GLAT fell short of generally accepted thresholds for reliable scores. This suggests that prior estimates of reliability may be inflated, as they were calculated from unfiltered datasets obtained from low-stakes administrations of the GLAT. The GLAT was shown to be a unidimensional test of AI literacy, which fit best with a two-parameter logistic (2PL) IRT model. Correlation analyses showed that GLAT scores did not correlate with how frequently students used AI, their attitudes towards AI, and their understanding of university AI policies. While scores did correlate with perceived AI competence, this correlation was small. Comparing GLAT scores across academic domains revealed that AI literacy scores do not generally differ across academic disciplines. Overall, the current study highlighted key areas for refinement regarding the GLAT. The study found that students frequently misestimate their own AI literacy, which is important as most existing measures of AI literacy are self-report tests. Further work should be done to understand these findings, and to refine the GLAT so that it can function well as an AI literacy test at all ability levels.