Creative Commons License

Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.

Date of Award

Spring 2019

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Department of Graduate Psychology

Advisor(s)

Dena A. Pastor

Deborah L. Bandalos

Christine E. DeMars

Monica K. Erbacher

Abstract

One type of context effect is a position effect, which implies parameters of an item are influenced by the position of the item on the test. Researchers often discuss two types of position effects: negative position effects and positive position effects (e.g., Albano, 2013; Debeer & Janssen, 2013). Items exhibiting negative position effects become harder when placed later on the test, whereas items exhibiting positive position effects become easier when placed later on the test. Researchers have primarily examined the underlying causes of position effects through an item or person perspective (e.g., Bulut, 2015; Kingston & Dorans, 1984; Qian, 2014). Researchers who adopted an examinee perspective on position effects exclusively studied the relationships among person variables and position effects. Researchers who adopted an item perspective on position effects exclusively studied the relationships among item variables and position effects. These two perspectives are limiting because they do not encourage researchers to consider the potential interactions among person variables, item variables, and position effects.

In this dissertation, I examined the underlying causes of position effects through an integrated perspective, where I studied the relationships among person variables, item variables, and position effects simultaneously. I conducted a true experiment in which I administered items from two low-stakes assessments in different order to two groups of examinees, examined the presence of position effects, and evaluated the degree to which position effects were moderated by different item (item length, number of response options, mental taxation, and graphic) and person variables (effort, change in effort, and gender). I modeled position effects and their relationships with item and person variables under the generalized linear mixed modeling (GLMM) framework.

On both assessments, I found items exhibited significant negative linear position effects on both assessments, with the magnitude of the position effects varying from item to item. Items became harder when placed later on the assessments but the extent to which they became harder differed slightly across items. Additionally, I found the position effects to be moderated by item difficulty and item length but not number of response options, mental taxation, or graphic. Easier and longer items were more prone to position effects than harder and shorter items; however, items varying in mental taxation, items containing a graphic, and items varying in response options were similarly prone to position effects. More so, I found examinee effort levels, change in effort patterns, and genders did not moderate the relationships among position effects and item features. Based on these findings, testing practitioners should be cautious about administering long or easy items in different order across forms and/or administrations.

Available for download on Wednesday, April 14, 2021

Share

COinS