A Pilot Study on the Impact of Teaching Assistant Led CS1 Study Sessions using Peer Instruction: An Undergraduate Honors Capstone Project

The aim of this research is to discover if a structured study session program for CS1 students led by Teaching Assistants using Peer Instruction can promote an environment in which students feel an increased sense of belonging and improve course material retention. This study investigates the efficacy of weekly study sessions by measuring student attitudes and content retention over the course of a single semester of the introductory CS1 course at a medium sized liberal arts university with approximately 8 sections of 30 students each.


List of Figures
James Madison University's Computer Science program strives to be a student-centered learning environment with a focus on creating a community for undergraduate success. National data reveals computer science has the lowest student retention rate compared to other STEM majors. The National Center for Women and Information Technology (NCWIT) has compiled a list of ways to retain students in Computer Science. In particular, NCWIT calls for collaboration indicate that "a sense of belonging, or a feeling of fit, is important for supporting student interest and persistence." One aspect of creating community is the department's longstanding commitment to provide undergraduate teaching assistants (TAs).
Traditionally, TAs provide one-on-one help in the classroom and also hold supplementary lab hours in the evenings to answer questions. As part of this honors project, we have developed a new program called "The Fourth Hour" to increase student retention. Led by TAs using Peer Instruction (PI), these weekly study sessions provide a structured review of introductory topics.
The aim of this research is to discover if weekly study sessions promote an environment in which students feel an increased sense of belonging and improved course material retention.
In the Fall 2019 semester, JMU offered ten sections of CS149, the introductory programming course, also known as "CS1" in the literature. Each section had approximately 30 students enrolled. Four TAs were chosen to lead one study session each week using the same lesson materials. Three attitudinal surveys were administered over the duration of the semester to collect data on student demographics, self-efficacy, and sense of belonging. Pre-and postassessment results were recorded to test student course material retention. Study session attendance was also collected to discern if there was a correlation with student demographics, self-efficacy, sense of belonging, and/or course material retention. Anomalies in the data and inconsistent attendance rates limited the statistical significance of our results. However, our qualitative analysis suggests that the study sessions had a positive impact on students. As a result, the CS department is planning to continue offering the Fourth Hour program. James Madison University strives to create a strong community in which, upon graduation, students feel prepared to "lead productive and meaningful lives" [1]. The University aims to follow six key values: Academic Quality, Community, Diversity, Excellence, Integrity, and Student Focus. By adopting these key values, JMU hopes to create an engaging environment which promotes student success.
Likewise, JMU's Computer Science department seeks to foster an environment that encourages the collective university's student-centered values. The Computer Science department promotes student learning through multiple outlets. All class sizes (with the exception of CS101) are capped at 30 students to create a personal environment. Clubs allow students to form cohorts and support systems. Diversity is promoted through active clubs, such as Women in Technology, where an underrepresented group in Computer Science is given a voice. A STEM career fair is offered to assist students in obtaining internships and employment prior to graduation. In conclusion, JMU's Computer Science program aims to build an environment in which students believe that they are a member of a community in which they can succeed.
Although JMU's program strives to create a community for students to flourish, Computer Science can be a challenging major-especially as several students have no coding experience prior to university. Across the United States, data exhibits that Computer Science has the lowest retention rate among STEM majors [2]. According to the National Science Foundation, from 2003-2009 it is estimated that the average retention rate for students in physical/computer/mathematical sciences at 4-year institutions was barely over 40%. But, overall retention in this category was lower than all other STEM majors. This raises the question: How can we increase retention rates in Computer Science?
To be formally accepted the Computer Science major at JMU, students must obtain a "B" average upon concluding their first two required Computer Science courses. CS149 and CS159 act as the foundation courses required by JMU students seeking to be accepted into the Computer Science Program. Upon completion of these courses, students will have the opportunity to apply to the major. Could more class time in CS1 courses be the answer?
Two years ago, JMU offered a 4-credit introductory CS1 course registered as CS139 in the course catalog in addition to the 3-credit equivalent course CS149. The 4-credit course allowed for students to meet one extra hour a week. But research showed that on average the 4-credit course did not reap additional benefits as students received similar grades as students in the 3-credit course. A study conducted by JMU faculty discovered the retention rate, defined as students continuing or graduating from the Computer Science program, was no different for students enrolled in CS139 versus CS149 [3].
As of 2017, CS139 has ceased being offered. It may be important to note that students would self-select into the section of CS1 that they wanted to take (CS139 or CS149) and a variety of factors may have prevented students from enrolling in CS139. For example, the 4-credit course may have been harder to fit into student schedules. Other science courses offered at JMU often offer labs and lectures independent of one another. It can sometimes be more straightforward for students to fit a 3-credit lecture class, and 1-credit lab class into their schedule. In addition, anecdotal evidence indicated that some students with prior experience still selected CS139, removing some of the benefits to those without experience being in a class with others of similar background.
Eliminating major requirements may seem to be a plausible way of increasing student retention, but previous results suggest this approach is unnecessary. Past evidence from a study performed by JMU faculty indicates that the grade a student earned in CS1.5 (equivalent to CS159 Object-Oriented Programming) affected student retention in CS2 (equivalent to CS240 Data Structures and Algorithms) courses [3]. This research informed the grade requirements for acceptance to the major as it indicates mastery of the introductory content is essential to student future success in completing the major. Students that fail to meet requirements traditionally must retake courses or do not complete the major [3]. Major requirements were therefore put in place with the intention of benefiting students. These lead TAs, to aid in student understanding of course concepts, introduced a weekly review session program. Weekly review sessions cover concepts taught the previous week during lecture. The aim was to assist CS1 students in retaining and comprehending course concepts, so they perform well on assessments. Review sessions at this time created by Lead TAs were unstructured-they did not utilize any learning pedagogy's facilitation guidelines. Despite this, students appeared to enjoy session availability, especially prior to exams.
Supplemental resources such as access to TA assistance can be helpful to students seeking extra assistance. But, it is also vital that students feel comfortable in seeking assistance when they are struggling to grasp learning material. Sense of belonging often aids in fostering student success [4]. The National Center for Women and Information Technology's (NCWIT) efforts stress the importance of student attitudes. Belonging becomes important especially for the success in underrepresented groups. The idea of belonging or "fit" can encourage student persistence [4]. Persistence is often vital in work heavy courses. Collaborative learning can form a sense of belonging among student groups. NCWIT recommends Peer Lead Team Learning (PLTL) as a collaborative setting in which peers aid student learning in workshops outside of traditional class hours [5]. Studies suggest, specifically in chemistry courses that PLTL can improve student attitudes. Students in a particular study report that they feel PLTL positively impacted their learning experience [6]. Positive feeling paired with collaboration can foster a sense of belonging. Creating an environment where students recognize they belong can be vital in student persistence within STEM majors.

Goals/Overview
The purpose of this study was to test the question if a structured study session led by Teaching Assistants (TAs) and attended regularly could increase CS1 material retention and attitudes in the Computer Science Program at JMU. To test this question, a structured CS1 study session curriculum was designed and implemented during Fall Semester 2019 to observe if a Lead TA (peer-run) study session program would be a beneficial addition to the JMU Computer Science Program. Four Lead TAs were chosen by the TA supervisor. Each of these TAs led one study session each week using the same lesson materials. Learning materials such as worksheets and slides in every study session were identical. Students completed a pre-and post-semester assessment that covered all of the material that was reviewed during the study sessions for the duration of the whole semester. All students registered in CS149 were invited to participate in the study. Students absent from all study sessions were used as a control group.
Students that consented to participating in the study were asked to complete the presemester assessment during the first week of their CS149 class. A week before finals, those same students were asked to take the post-semester assessment. Results from pre-and post-semester assessments serve as the measure of CS1 material retention. CS1 student attitudes were measured three times over the course of the semester during week 2, week 8, and week 14. Attitudes were calculated using student survey responses. Attitudes and CS1 material retention were compared upon completion of the study to analyze the effectiveness of the introduction of a structured study session program.

Background: Choosing a Teaching Pedagogy
Several teaching pedagogical approaches were under consideration for this project. It was determined study sessions would be designed to follow the teaching methodology of Peer Instruction but Peer Led Team Learning, Process Oriented Guided Inquiry Learning, and Pair Programming were also investigated. Note that only active learning methodologies were considered because there is strong empirical evidence that active learning produces better learning outcomes [7]. An outline of costs and benefits is described in Table 1.1.

Peer Led Team Learning
Peer Led Team Learning (PLTL) breaks students into groups of up to eight students. Groups are led by a peer leader or TA. Worksheets are given to teams to complete over a given workshop period [8]. PLTL benefits include "significant gains in performance, retention, perseverance, and student attitudes and opinions" in STEM areas [9]. The resources PLTL requires are significant: one TA per group of eight students, one classroom per group, and at least two hours of workshop time each week. Due to lack of resources, PLTL was out of scope for this experiment. Instead, PLTL materials and attitude assessments will act as references and a comparison tool for this study. Peer Instruction was instead chosen for this study for its ability to allow students to work collaboratively in small groups like PLTL but without the need for an extensive amount of resources. As described below, there is evidence that PI also increases student performance and retention. -Groups must be chosen well -Groups attendance is imperative -Projects must be group projects -Technology must be accessible Peer Instruction -Small groups -Increase performance -Increase student retention -Promotes collaboration -First-time professors generally make good PI questions -Increase in student retention is immediate -Allows for flexibility -Groups can change each class period -Technology must be accessible -Groups must feel encouraged to discuss "why" they chose answer -PI questions must be well designed

Process Oriented Guided Inquiry Learning
Process Oriented Guided Inquiry Learning (POGIL) requires students form groups of approximately four students. Each student is designated with a specific role. Roles often rotate each class period. Each class, students are given a packet to complete as a team. Professors observe, guide, and aid students that have any questions. POGIL was considered for this experiment due to its popularity within JMU's Computer Science Department. Several professors use POGIL instead of lecture. Team learning can reduce the feelings of isolation and competitiveness which is essential for retaining non-traditional students [10]. Although POGIL has positive material retention results, often professors teaching the course have to go through training before starting [9]. Perfecting the style of teaching using POGIL requires experience. POGIL also encourages groups to contain the same students throughout the semester. POGIL was unrealistic for this experiment as study session attendance varies from week to week and students were not required to attend, whereas in a traditional POGIL setting attendance is required. POGIL is practiced and designed more for a traditional classroom setting versus a study session program.

Pair Programming
Pair Programming was another consideration for this project. This style of learning allows two students to pair up and collaborate on a coding assignment. One student is allowed to code at a time while the other observes. Students acting as the observer often discover coding mistakes early. The observer is also tasked to provide insights in forming solutions [11]. Pair-Programming has shown positive overall results in increasing retention and performance, for example in studies at the University of California at Santa Cruz (UCSC) [12]. JMU currently has TAs that help with coding concepts in the evenings. In addition, the study sessions focus on CS1 concepts as opposed to coding.

Peer Instruction
Peer Instruction (PI) was chosen as the basic guideline for the study sessions as data shows that even for first-time professors, the results of using PI are still significant in helping students retain material [13]. Unlike a regular lecture, PI tries to increase student retention of material by having students work together to answer questions. First, students will be asked to answer a question individually via an online quiz tool. Then, students will break off into groups to discuss their answers and collectively come up with a group answer. When students form groups after initially answering the question themselves, overall responses increase in accuracy [14]. Afterwards, students are asked to work individually to answer another question that is similar in content to the first. Findings show that individual scores increase for the second question [14]. Because of the positive data that PI reflects and the fact that first-time professors are able to perform well using PI (students are able to successfully benefit and retain information from professors who use PI for the first time), PI was chosen as the methodology used for the study sessions.

Related Work
Prior work in the field that inspired this project includes research on Peer Led Team Learning, papers on Peer Instruction, and the unstructured study session program developed by lead who have little to no programming experience or are in minority groups [15]. The class was offered as an additional one credit optional course. The class is "supplemental learning, not remedial". The inital plan had been to design the study session program as a Pass/Fail one credit "did you show up" course. Due to lack of resources and shortage of time, implementing PLTL as a one credit course became infeasible. But, it continually inspired the idea of the study session program as a form of increasing inclusively in minority groups and providing aide for students struggling in CS1 courses. PI (Peer Instruction) was considered early on as a possible teaching pedagogy. An instructor who uses Peer Instruction in several of their courses recommended several PI resources.
A journal entry written by Zingaro and Porter aims to answer the question if instructor intervention after students answer questions in a PI classroom environment is beneficial [13].
According to this study, the answer is yes. Every student no matter how good of a student they are not only benefits from a PI environment, but succeeds even more when an instructor intervenes (discussion after answers are given). This paper was used to model implementation of PI questions for this study.
Taylor et. al. aim to answer the question if professors are correctly using PI, and when do they become proficient at coming up with PI questions [14]. According to the study, professors using PI come up with decent questions in the first year of using PI in their class.
But, within 2 years, their questions become 2x better. This article shows it is doable for a first time instructor to be able to come up with good questions. This article aided in choosing PI as the teaching pedagogy for the study session program as it showed promise TAs would be able to implement PI effectively within the first year.
Dillon and Slattery [16] describe a curriculum revision at Michigan State University.
Attempting to increase CS retention, confidence, and sense of belonging in an MSU CS1 course, a support class was created as part of larger "efforts to broaden diversity." The support class nicknamed Computational Thinking Lab (CTL) was designed as a 1 credit supplementary course for students enrolled in the CS1 course at MSU. The CS1 course at MSU is a 4-credit course with 120 student lectures and 25 student labs. Students in CTL met once a week for 2-hours. CTL sessions were led by undergraduate peer leaders following PLTL teaching pedagogy [17]. Results showed students enrolled in CTL experienced an increase in sense of belonging over the course of the semester, had higher grades on average than the non-CTL group, and "none of the CTL students dropped or withdrew" [16]. The experiment performed at MSU published in 2018 depicts the type of program this study originally considered. Due to lack of resources a 1-credit course and PLTL were infeasible.
But, a similar program was created using PI given the resources at JMU. It is important to note that JMU CS1 equivalent courses are relatively small compared to MSU with a max of 30 students in each CS1 section at JMU. Four TAs were hired to lead the study sessions. Three TAs acted as primary study session TAs while one TA acted as a secondary study session TA. Primary TAs would typically act as presenter and observer both once a week. The secondary study session TA was asked to act as a presenter or observer when one of the primary TAs was unavailable due to illness or schedule conflict. Two TAs were always present during the study sessions. One TA acted as the presenter and the other the observer. The presenter's responsibilities included leading the session by reviewing slides with students, answering student questions, and working out CS1 problems. The observer was responsible for collecting study session attendance, ensuring all CS1 concepts were described correctly by the presenter, and answering questions the presenter was unsure or unable to answer. Slideshows, activities, and worksheets created by a Lead TA were reviewed by a faculty supervisor prior to being presented to CS1 students during the Fourth Hour study sessions.
Slideshows were shared amongst study session TAs through Google Drive ensuring all three study sessions offered each week contained identical supplementary learning materials. A typical study session would begin with asking students to introduce themselves as many students came from various sections of CS1. Each week a slideshow was used to guide the study session lesson plan. Slideshows included three Peer Instruction Questions, cooperative CS1 activities and problems for groups of 2-3 students to solve, and typically reviewed chapter vocabulary. Sessions encouraged students to work together to solve various problems. Music was often played to encourage a friendly relaxed environment.

Session Schedule
The Fourth Hour schedule can be found in tabular form on the following page. All materials used during study sessions are publicly available via https://github.com/mGilbert15/ The-Fourth-Hour. Note that study sessions were not offered the week of Thanksgiving Break as James Madison University Campus and Residential Halls were closed. The first week of the semester served as a crucial time for advertising the Fourth Hour. Flyers, and emails were sent to students registered in CS149 during Week 1 of the semester advertising the Fourth Hour.

Implementing Peer Instruction
When implementing Peer Instruction (PI), we chose the free online tool Socrative. Within the Computer Science Department at JMU, several professors have utilized Socrative in the classroom. Socrative has been used as a tool when implementing PI and gaining student interaction in class. Wash [18] found that students reported strong agreement that using an online tool such as Socrative increased their class participation, helped them receive feedback on their performance, and aided in creating positive interactions with their peers and/or professor. Socrative is easily accessibly as it is free to students, a classroom code is used to sign in as opposed to creating a personal account, and platform friendly: usable on mobile devices and laptops [19].
At the beginning of the semester a Socrative classroom was created for the Fourth Hour.
Prior to the first PI question, students were requested to access the Socrative classroom using  the PI quiz allowing students to use software anonymously and questions were teacher-paced allowing TAs to allow 1-2 minutes per-question based on student interactions. The first time a question appeared, students were asked to complete the multiple choice or true-false question independently. After 1-2 minutes, the TA would move on to the next question (which was identical to the first question) but students were now allowed to talk in groups of 2-3 students to discuss their answers. Students were encouraged by the presenting TA to share why they chose a particular answer during the first round of questioning within their groups. Students were permitted to change their answer during the question-repeat. After students finished answering the question-repeat TAs shared the percentages of students who chose each particular answer. Using the whiteboard or slideshow the presenting TA would go over both the correct answer and why the other answers were incorrect. Over the course of the study session, two more PI questions would be administered using the identical process described above.

Creating Peer Instruction Questions
Our • T1: Students misunderstand the relationship between language elements and underlying memory usage.
• T2: Students misunderstand the process of while loop operation.
• T3: Students lack a basic understanding of the Object concept. Most of these misconceptions are self-explanatory to Computer Science faculty, with a couple exceptions. We will refer to "semantics to semantics", which refers to a student misunderstanding variable declaration by applying real world semantics to the process, as "semantics to variable declarations". And we will refer to "primitive no default", which refers to a student misunderstanding that instance variables of a class have no default value, as "instance variable no default".

Demographics
Demographic data was collected from students in Survey 1, which included student race, gender and prior programming experience, among other information.

Evaluating Content Retention
To evaluate student content retention over the course of a semester the core assessment scores were analyzed. Calculated CS1 student pre-CA (core assessment) score (out of 100) as well as CS1 student post-CA (core assessment) score (out of 100). Normalized learning gain was calculated to evaluate how much additional information students learned over the course of the semester using the following equation: Students with a low normalized learning gain score had less learning gain than students with higher scores. Several students had a negative learning gain score, indicating that they scored higher on the pre-CA than the post-CA. Negative scores could be attributed to good guessing on the pre-CA. A statistical online resource for R was referenced as a guide for testing significant difference on non-normal data [21] as well as normal data [22].

Evaluating Based on Attendance
This section discusses the question: Is there a significant difference in learning retention between students that attended Fourth Hour and students who did not attend Fourth Hour?
To answer this question normalized learning gain scores were compared based on if a student attended Fourth Hour at least once versus never.  To test for significant difference, first the data was evaluated in the programming language R using a Shapiro-Wilk test to evaluate normality with a significance-level of 0.05. Neither the Never-Attended category (p = 1.474e-06) nor the Attended category (p = 0.007993) exceeded the significance level p ¿ 0.05, so the data was not considered normal. Due to this lack of normality, we used a Wilcoxon Rank Sum test to determine that the difference was statistically significant (W=2761, p=0.0004443).
This result suggests that there is statically significant difference in learning gain in students who attended the Fourth Hour versus students who never attended the Fourth Hour.
Students who did not attend Fourth Hour had higher material retention than students who did attend the Fourth Hour. One plausible explanation is that students who attended the Fourth Hour were more likely to be struggling students seeking help.

Evaluating Students with no Previous Coding Experience
This section discusses the question: Is there a significant difference in learning retention between students who had no previous coding experience who attended Fourth Hour that had High frequency attendance vs. Low frequency attendance?
Similar to the previous question, normalized learning gain scores were compared based on attendance frequency. Data for students that lacked completion of either the pre-CA or post-CA or had indicated prior programming experience were not included in this analysis.

Evaluating Sense of Belonging
Students who participated in the study were asked to take three surveys over the course of the semester. These surveys were not identical, but five questions appeared on all three surveys.
These questions were preceded by the statement, "In this computer science class. . . ": 1. I feel that I belong to the computer science community.
3. I feel like an outsider. 4. I try to say as little as possible.

I trust my instructors to be committed to helping me learn.
The purpose of these questions was to track student sense of belonging throughout the semester. But, several students did not complete all three surveys. As a result, analyzing individual sense of belonging change throughout the semester became difficult. In the end, it was decided the last survey would be used to evaluate student sense of belonging at the end of the semester. 204 students completed the third and final attitudinal survey administered on the 14th week of the semester.
Questions were presented in Likert Scale format. To analyze sense of belonging, each item was given a numerical value. Positive items (1,2,5) were given the numerical values: • strongly disagree : 1 • disagree : 2 • neutral : 3 • agree : 4 • strongly agree : 5 Negative items (3,4) numerical value scales were reversed (i.e., strongly disagree : 5, strongly agree : 1). For each student, sense of belong was determined by calculating the mean score among the five items: Sense of Belonging scores closer to 1.0 will correspond to low student sense of belonging, while scores closer to 5.0 will correspond with high sense of belonging.

Based on Demographics
This section discusses the question: Are there differences in responses to the attitude survey based on demographics?
Based on mean scores, there appears to be a difference in sense of belonging based on racial demographics. Due to low sample numbers in each race category, statistical testing was difficult. Mean scores were compared to understand student sense of belonging based on race. A box plot rendered in R Figure 3.4 depicts sense of belonging in students at the end of semester in CS149 students based on racial demographics. There appears to be a difference in sense of belonging based on racial demographics.  Hour by the mean score of students in that racial category that never attended Fourth Hour.
Positive values in the last column indicate students that attended Fourth Hour had a higher sense of belonging than those that never attended while negative values indicate the opposite.

Based on Attendance
This section discusses the question: Is there a relationship between study session attendance and attitude responses?
Due to small sample size, specifically of students with High Frequency Fourth Hour attendance, it is difficult to prove significant difference statistically.

Primary Results
In summary, this study found that:

Students who didn't attend Fourth Hour had a greater sense of belonging and learning
gains. This result was found to be statistically significant. Future research will explore qualitative gains of students who attended.
2. Students in the minority bracket with no prior programming experience were more likely to attend Fourth Hour than those with experience.
3. Around half of the Black or African American and Hispanic or Latina/Latino populations in CS149 attended the Fourth Hour at least once.

Other Results and Observations
Through discussions with students and teaching assistants, we also found that:

Students who attended Fourth Hour frequently felt encouraged. Attendees
talked about what to do later in the major and whether or not to repeat CS149 to be more successful later. Students asked for TA advice and those struggling in the course that intended to repeat expressed attending sessions helped in understanding topics.

5.
Students who attended Fourth Hour frequently reflected learning material.
For example they improved performance on PI questions and spoke up that they now "got it". Several students expressed review in Fourth Hour helped them prepare for CS149 assessments. Some indicated attending had a large impact on grade received and aided in improving score on second midterm.
6. Students encouraged each other, displayed teamwork, and praised team mates for correct answers. Several students came from different CS149 sections.
They often shared techniques they learned in their section to aid in answering questions.
Students worked together and often displayed encouraging behavior.

Lessons Learned
Finally, we summarize several lessons learned and key insights to inform future work.
a. Motivation is key Eliminating cost and increasing value of study sessions may help increase Fourth Hour attendance [23]. Publicity, strategies for making it easy for students to attend, incentives for attendance, encouraging students to sign up during CS149 class time, having TAs visit class, having students visit space where sessions are held, and encouraging students to bring friends can be used as techniques to increase student attendance in the future. Extra credit opportunities or snacks/dinner offered during Fourth Hour sessions could be used as tools to increase value of attending Fourth Hour sessions.
According to recent work by Dr. Kenn Barron, cost is a significant factor in motivating students. Cost should be added to the Expectancy-Value model as "a student needs to be free of costs in order to demonstrate motivated behavior" [23]. Spikes in Fourth Hour attendance prior to exams depict Fourth Hour value increased for most student prior to exams. Examples of cost for the current Fourth Hour program included time, and fear of self-image (e.g appearing to be behind in class, or not as knowledgeable as other students). Eliminating costs using techniques listed above in addition to instilling a growth mindset in students may increase attendance. Higher attendance numbers are needed both to improve impact and to figure out significance.
b. Struggling students will be the ones to choose to get help and so learning achievements and sense of belonging will be depressed for Fourth Hour compared to students with previous experience who are not underrepresented and find it easy to think of themselves as computer scientists from the beginning.
c. Accounting for previous experience does not completely compensate for item "b." above.
d. Consideration for transforming the study session program into a "1 credit hour support class" similar to the Computational Thinking Lab (CTL) program at MSU would promote consistent attendance [16]. Transforming Fourth Hour into a 1-credit course "recommended to students with limited prior experience or to women or minority groups" similar to CTL could help increase retention and sense of belonging in targeted groups [16]. To reduce resources required for implementation of CTL, the adopted 1-credit course would use materials created for the Fourth Hour, continue to use the PI teaching pedagogy, and utilize the same TA resources in the past. Students would receive course credit for attending Fourth Hour which may increase its value and lower its cost for several students. "When students value something" such as gaining course credit,"they are more likely to do the task" which could increase motivation for students to attend Fourth Hour [23].

A.2 Attitudinal Surveys
Each survey was one sheet of paper front and back. Surveys were administered and gathered by the data collector during student class time.
• Part 1: Demographics, Experience, Confidence Prior to taking this course, have you had any programming experience? (e.g., Java, Python, Scratch)

Yes No
If yes, mark all that apply. Please also write the programming language(s) to the right of each course.  Considering the difficulty of this course, the teacher, and my skills, I think I did well in this class.
Please choose how much you agree with each statement.

Strongly Disagree
Disagree Neutral Agree Strongly Agree I find working as part of a team in courses to be a valuable experience.
Teams in courses work well together.
I find teamwork to be a productive use of course time.
I find that teams help me learn course material more than if I just studied alone.
I learn more in courses where I have been a member of a team.
I find being part of a team improves my course grades.
I find that working with a team helps me develop skills in working with others.
I find that working with a team helps me develop more respect for the opinions of others.
I find that working with a team enhances my sense of who I am.
I find that being on a team helps me become better at problem solving.
Being part of a team discussion improves my ability to think through a problem.
I feel that working in teams improves my critical thinking skills.
I find that teams make good decisions.
In this computer science class . . .

A.4.1 Learning Retention Graphs
The following code corresponds to the creation of Figures 3.2 and 3.3 given the data from this study. The statistical analysis process is included.
1 # reads data from csv install . packages ( " readr " ) 3 library ( " readr " ) data <-read _ csv ( " Honors Thesis / scripts / Fall2019Data . csv " ) 5 attendance <-read _ csv ( " Honors Thesis / scripts / Attendance . csv " ) 7 # color constants for graphing color1 <-' sandybrown ' 9 color2 <-' royalblue3 ' cbbPalette <-c ( " #000000 " , " # E69F00 " , " #56 B4E9 " , " #009 E73 " , 11 " # F0E442 " , " #0072 B2 " , " # D55E00 " , " # CC79A7 " ) 13 # calculate pre -test score percentage ( out of 10 points ) data $ ScorePre <-( data $ 'Pre -Test ' / 10) * 100 # constants for frequency highFreq <-" High Frequency " 69 lowFreq <-" Low Frequency " noFreq <-" No Frequency " 71 # assigns frequency based on counts 73 data $ F H A t t e n d a n c e F r e q u e n c y <-counts [ data $ newID ] data $ F H A t t e n d a n c e F r e q u e n c y [ data $ fhAttendance == 0] <-noFreq 75 data $ F H A t t e n d a n c e F r e q u e n c y [ data $ fhAttendance > 0] <-lowFreq data $ F H A t t e n d a n c e F r e q u e n c y [ data $ fhAttendance >= 4] <-highFreq 77 # creates tables based on Fourth Hour attendance frequency 79 h i g h F r e q u e n c y A t t e n d a n c e <-subset ( data , data $ fhAttendance >=4) l o w F r e q u e n c y A t t e n d a n c e <-subset ( data , data $ fhAttendance > 0 & data $ fhAttendance < 4) 81 n o F r e q u e n c y A t t e n d a n c e <-subset ( data , data $ fhAttendance == 0) 83 # creates a table of students that indicated no previous programming experience 85 dataNoExperience <-subset ( data , data $ P1Q05 == " No " ) 87 # Figure 3.5 # Fourth Hour Attendance Frequency ( no programming experience ) 89 library ( " ggpubr " ) p = ggboxplot ( data = dataNoExperience , x = " F H A t t e n d a n c e F r e q u e n c y " , y = " KnowlegeGain " , 91 color = " F H A t t e n d a n c e F r e q u e n c y " , palette = cbbPalette , ylab = " Normalized Learning Gain " , xlab = " CS149 Students with No Experience Fourth Hour Attendance " ) 93 # adds legend to graph 95 p + labs ( color = " Fourth Hour Attendance " ) 97 # taking out students with prior programming experience highFreqNoExp <-subset ( highFrequencyAttendance , h i g h F r e q u e n c y A t t e n d a n c e $ P1Q05 == " No " ) 99 lowFreqNoExp <-subset ( lowFrequencyAttendance , l o w F r e q u e n c y A t t e n d a n c e $

A.4.2 Sense of Belonging Graphs
The following code corresponds to the creation of Figures 3.4 and 3.5 given the data from this study.

110
# creates a boxplot of sense of belonging based on Fourth Hour attendance frequency 112 p = ggboxplot ( data = data , x = " F H A t t e n d a n c e F r e q u e n c y " , y = " B3Results " , color = " F H A t t e n d a n c e F r e q u e n c y " , palette = cbbPalette ,