Creative Commons License

Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.

Date of Graduation

5-6-2021

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Department of Graduate Psychology

Advisor(s)

S. Jeanne Horst

Abstract

In educational contexts, students often self-select into specific interventions (e.g., courses, majors, extracurricular programming). When students self-select into an intervention, systematic group differences may impact the validity of inferences made regarding the effect of the intervention. Propensity score methods are commonly used to reduce selection bias in estimates of treatment effects. In educational contexts, often a larger number of students receive a treatment than not. However, recommendations regarding the application of propensity score methods when the treatment group is larger than the comparison group have not been empirically examined. The current study examined the recommendation to recode the treatment and comparison groups (i.e., two types of treatment effect coding; Ho et al., 2007).

A simulation study was conducted to examine the performance of three propensity score methods (nearest neighbor matching, nearest neighbor matching with a 0.20 SD caliper, and generalized boosted modeling), using two coding methods (ATT and ATC) when the treatment group was larger than the comparison group. Additionally, three treatment sample sizes (200, 600, 1,000), three treatment to comparison group ratios (2:1, 4:3, 1:4), and four true treatment effects (Cohen’s d of 0, 0.20, 0.50, 0.80) were simulated.

For nearest neighbor matching with a 0.20 SD caliper, adequate group covariate balance and low bias in the estimated treatment effect were observed across both coding methods regardless of which group was larger. In contrast, for generalized boosted modeling and nearest neighbor matching, group covariate balance and bias in the estimated treatment effect differed across coding method. When the treatment group was larger than the comparison group, ATC coding resulted in better group covariate balance and lower bias than ATT coding. However, ideal balance was not obtained on all covariates, and bias in the estimated treatment effect was high for generalized boosted modeling and nearest neighbor matching. In sum, when the treatment group was larger than the comparison group, coding method did not matter for nearest neighbor matching with a 0.20 SD caliper. Conversely, for generalized boosted modeling, ATC coding performed better than ATT coding. Nearest neighbor matching did not perform well regardless of coding method.

Available for download on Friday, April 08, 2022

Share

COinS