Chapter 4: Results

Chapter 4 Results

Data Analysis

Analysis was conducted to examine Research Question 1 (Does the use of high quality instructional videos have an effect on the intervention learning of mathematics students?) and Research Question 2 (Is it more effective to use direct instruction or instructional videos to address student needs?) by analyzing the pre- and post-assessments for knowledge gained on each topic as well as looking at the teacher log to track students’ progress. Additionally, the attitudinal survey was examined to see how students’ attitudes and beliefs have evolved from before the study to after the study. Data was collected from the assessments, the topics tests, the attitudinal surveys, and the interviews. The assessments and tests were scored out of 100 points.

Pre- and post-assessment:

The pre and post assessment measured growth for 16 mathematical skills over the duration of the study. The pre-assessment results showed deficits in many key topic areas for middle school students. Out of these 16 skills, students received interventions in five topics during this study. Only the data from these five topics were included in the study. The average score on the pre-assessment was 3.95 percent. The highest average score was 12.5 percent in the area of mixed numbers and improper fractions. The highest scores on the pre-assessment were 6G2 with 25 percent correct with 6G1 scoring 6.6 percent. The rest of the students earned scores of zero on the pre-assessment. Overall, from the pre-assessment to the post-assessment there was an average growth of 60.93 percent.

 

 

The assessment measured students’ knowledge before the study began and after treatment had been received. Students who received treatment for 9 weeks have are displayed in Table 1a, while students who received treatment for 12 weeks are displayed in Table 1b [See Table 1a and 1b].

Table 1a: Study Pre- and Post-Assessment (9 Weeks)

 

Table 1b: Study Pre-and Post-Assessment (12 Weeks)

  Pre- and Post- Topic Tests

Student growth in individual lessons was measured through the pre and post topic tests. These tests were measured by comparing control treatment to experimental treatments. The results were inconclusive since many students started at zero knowledge. The growth was equal (meaning students in control and experimental lessons had the same average post-test score as well as the same amount of growth from pre-test to post-test) for three of the lessons: prime factorization, least common multiple, and greatest common factor. With one of these lessons, least common multiple, the students had equal post-test scores, but the experimental group had 10.5 more points of growth. On two topics (mixed numbers and improper fractions, which were measured on the same pre and post-test) students in the control group had higher average post-test scores by twelve points, and the control group had a little over 21 more points of growth.

The tests were administered during the study before and after each lesson. They demonstrated the growth in student knowledge immediately after the lesson was completed. The averages for each lesson for the control and experimental treatments are shown here. The lessons are color coded: experimental lessons are in orange and control lessons are in blue. [See Table2, See Appendix G for specific student results on all topic tests.]

Table 2. Pre- and Post- Topic Tests Averages:

 

 

Mixed Numbers and Improper Fractions

 

 

Prime Factorization

 

 

Least Common Multiple

 

 

Greatest Common Factor

 

 

Pre

Post

Pre

Post

Pre

Post

Pre

Post

Experimental

18.25

88.675

0

100

0

100

0

100

Control

8.25

100

0

100

10.5

100

0

100

Average

13.25

94.3375

0

100

5.25

100

0

100

 

On the study long assessment, results were slightly different (See Table 3). On three of the five topics, students who received experimental treatment scored higher on the post-assessment and therefore better in long-term retention, than those who received control treatment. Only on one topic, prime factorization, did students score better with the control treatment; and with the fifth topic, the outcome was equal for experimental and control groups. [See Table 3, See Appendix H for specific student outcomes on the assessment].

Table 3. Pre and Post Assessment Results by Lesson

 

 

Mixed Numbers

 

 

Improper Fractions

 

 

Prime Factorization

 

 

Least Common Multiple

 

 

Greatest Common Factor

 

   

Pre

 

 

Post

 

 

Pre

 

 

Post

 

 

Pre

 

 

Post

 

 

Pre

 

 

Post

 

 

Pre

 

 

Post

 

Experimental

25

75

8.25

75.5

0

74.83

0

87.5

0

75

Control

0

50

0

16.5

0

83

0

37.5

0

50

Average

12.5

62.5

4.125

46

0

78.915

0

62.5

0

62.5

 

When comparing the post-test (which measured more immediate learning) results with the post-assessment (which measured the retention of learning over a longer time period), the students’ scores were higher on the post-test than on the post-assessment as student retention decreased over time. Students scored over 35.52 points better on the post-test than on the correlated lesson topic area on post-assessment. However, when breaking it down to whether students received control or experimental treatment on the topic, overall, students who received control treatment scored 45.8 points lower on the post-assessment than on the post-tests. Students, who received experimental treatment however, scored only 21.68 points lower on the post-assessment when compared to the post-test. This demonstrated that while retention for both decreased over time, it decreased considerably less for those who received the experimental treatment than those who received the control treatments in specific lessons.

Attitudinal Survey

The Pre-Attitudinal Survey (see Appendix A) measured student’s previous experience with SRBI and instructional videos as well as what they liked best, liked least, and would like to change. Both were measured using a five point Likert-scale with 1 by strongly disagree and 5 strongly agree, 3 neither agree nor disagree; and 0 not applicable/not sure; as well as using short answer responses. Three students had watched instructional videos in educational settings before this study. Of these three students, all believed watching the videos helped them learn. Four students either agreed or strongly agreed to the statement that past experiences with SRBI had increase their math skills, while one student neither agreed or disagreed and three students were not sure or it was not applicable (not all students had received SRBI in the past) (see Table D1 in Appendix D).

The Post-Attitudinal Survey measured students’ current SRBI experience, their experience with instructional videos and their preferences for either instructional videos or direct instruction. All students either agreed or strongly agreed that their experiences with SRBI thus far this year had been positive. All but one student also agreed or strongly agreed that SRBI had helped to increase their math skills. The outlier was not sure. Four students felt watching the instructional videos helped them to better understand math concepts. Two students neither agreed nor disagreed, one student disagreed, and the last student was not sure. Five students preferred instructional videos over direct instruction. Two students preferred direct instruction over instructional videos, and one student was not sure which they preferred (see Table D3 in Appendix D).

Focus Group Interview

The focus group interview asked twelve questions [See Appendix B]. (Appendix C provides a full transcription of the interviews.)The first two questions asked about their perceptions of direct instruction, They generally liked the direct instruction followed by practice problems but they did not like that it was challenging and took a long time. Questions three and four asked about the video instruction on Khan Academy. The students liked that Salman Khan showed how to do the work and solve the problems with detailed instruction, using drawings and pictures

The fifth question addressed KAC website layout, and design and delivery. Students were equally split between thinking the website was easy to use and thinking that it was hard to use – they struggled with the set up and finding the topic they were looking for. In regards to the videos, students felt they were easy to use, with the colors helping to guide them.

The sixth question addressed the use of worksheets. The students liked taking the notes but felt that Salman Khan talked too fast. They adjusted by rewinding or pausing the video. The Five in a Row requirement on Khan Academy had mixed feedback. Some students liked Five in a Row because it was fun, helpful, challenging and guided them through the learning process. Others found it to be more of a challenge and preferred just watching the video. Two students would have preferred a composite score rather than the Five in a Row.

The eighth question asked if students would rather watch the video, use the practice section (5 in a row), or do both. Most students chose both, although two chose just the video part, and a third student chose neither. Students did not like the hints, feeling they were unhelpful or just gave them the answer.

The next three questions (9, 10, and 11) asked about the Tier III intervention process. Some students were unsure whether, or why, they liked it, while other students mentioned the advantage of completing homework, and becoming stronger in math. They felt the extra time gave them an advantage in math class because material was covered in more depth and over more time than in class, as well learning multiple strategies to solve problems. They wanted to change such things as the time they came (missing specials), avoiding learning about fractions, being provided with a snack, and being able to complete other homework. Two students did not want to change anything. One student did not like the overlapping topics (question 11) because he did not like having material reviewed. Another student sometimes found this helpful and sometimes found it confusing. Most students found the overlap helpful because it made it easier in math class. They had a chance to learn more and they learned multiple approaches to solve problems.

The twelfth, and last, question asked if students preferred either Khan Academy or direct instruction. The student responses were split down the middle, with four students preferring direct instruction and four students preferring Khan Academy (see Appendix C).

By comparing answers in the interview to answers in the attitudinal survey, consistency in responses could be checked. Comparing responses to interview question 9 and post-attitudinal survey question 6, which both asked what they liked best about receiving SRBI interventions, showed that answers were consistent except for one student who said “I don’t know” in the interview but wrote “gives your support and help with math” on the survey. Comparing questions about what they liked least about SRBI interventions (question 10 in the interview and 7 and 8 on the attitudinal survey), three students’ answers were consistent but the rest were not. Some students left the question on the attitudinal survey blank while others simply wrote different answers compared to what they said in the interview. One frequent response on the survey but not in the interview was that students were missing their favorite special to come to SRBI. This may have come up more on the attitudinal survey rather than the interview because it was mentioned by one student during the interview and other students, upon reflection, agreed.

Comparing question 12 in the interview to questions 4 and 5 on the attitudinal survey (preference for Khan Academy Coach or direct instruction) showed consistency in whether students preferred direct instruction or instructional videos. Six of the student’s answers were consistent from interview to survey. One student said on the attitudinal survey that they did not know which they preferred. In the interview they said they preferred both, but direct instruction during school. Another student in the interview preferred direct instruction. In contrast, on the attitudinal survey he strongly agreed with preferring instructional videos over direct instruction. This student repeatedly mentioned that he did not like the narrator’s voice in the video. He also expressed how he did not like Five in a Row, since getting one wrong, meant having to do five more. He preferred to do the whole test at once then be graded at the end. He had mentioned how he enjoyed the colors and that he would watch the video if the voice was different.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s