Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
Date of Graduation
5-6-2021
Document Type
Dissertation
Degree Name
Doctor of Philosophy (PhD)
Department
Department of Graduate Psychology
Advisor(s)
S. Jeanne Horst
Christine E. DeMars
Heather Harris
Dena A. Pastor
Abstract
In educational contexts, students often self-select into specific interventions (e.g., courses, majors, extracurricular programming). When students self-select into an intervention, systematic group differences may impact the validity of inferences made regarding the effect of the intervention. Propensity score methods are commonly used to reduce selection bias in estimates of treatment effects. In educational contexts, often a larger number of students receive a treatment than not. However, recommendations regarding the application of propensity score methods when the treatment group is larger than the comparison group have not been empirically examined. The current study examined the recommendation to recode the treatment and comparison groups (i.e., two types of treatment effect coding; Ho et al., 2007).
A simulation study was conducted to examine the performance of three propensity score methods (nearest neighbor matching, nearest neighbor matching with a 0.20 SD caliper, and generalized boosted modeling), using two coding methods (ATT and ATC) when the treatment group was larger than the comparison group. Additionally, three treatment sample sizes (200, 600, 1,000), three treatment to comparison group ratios (2:1, 4:3, 1:4), and four true treatment effects (Cohen’s d of 0, 0.20, 0.50, 0.80) were simulated.
For nearest neighbor matching with a 0.20 SD caliper, adequate group covariate balance and low bias in the estimated treatment effect were observed across both coding methods regardless of which group was larger. In contrast, for generalized boosted modeling and nearest neighbor matching, group covariate balance and bias in the estimated treatment effect differed across coding method. When the treatment group was larger than the comparison group, ATC coding resulted in better group covariate balance and lower bias than ATT coding. However, ideal balance was not obtained on all covariates, and bias in the estimated treatment effect was high for generalized boosted modeling and nearest neighbor matching. In sum, when the treatment group was larger than the comparison group, coding method did not matter for nearest neighbor matching with a 0.20 SD caliper. Conversely, for generalized boosted modeling, ATC coding performed better than ATT coding. Nearest neighbor matching did not perform well regardless of coding method.
Recommended Citation
Perkins, Beth A., "Does coding method matter? An examination of propensity score methods when the treatment group is larger than the comparison group" (2021). Dissertations, 2020-current. 51.
https://commons.lib.jmu.edu/diss202029/51
