Outcomes

Student Outcomes

Types of ISAT Studies

Previous Page Next Page

Citywide Math Specialist Studies

Sidebar: What did these in-school coaching studies tell us?

This CMSI research examined the impact of Citywide Math Specialists on ISAT math scores. The Citywide Math specialist studies focused on only one component of one program: leadership training on the use of Everyday Mathematics. Still, these analyses introduced "Theory of Action" variables (described below) into the broader CMSI research lexicon and their findings are strong indicators of the positive effect of the CMSI.

In school year 2005-2006, CMSI assigned Citywide Math Specialists ("CMS") to work in a subset of CPS schools. Their role was to assist teachers with implementation of the CMSI-endorsed instructional material. In the same year, some Everyday Mathematics ("EDM") teachers enrolled in a specialized EDM Leadership Training course. The citywide math specialist studies examined the impact of CMS assignment, EDM program implementation and EDM Leadership Training on two sets of variables – "Theory of Action" variables and "Performance" variables.

The following is a detailed description of the variables and analyses used in these studies. The analyses examined five "Theory of Action" independent variables. These included: 1) average number of PD hours registered for by teachers at a school; 2) average number of PD hours attended by teachers at a school; 3) Ratio of PD courses attended to courses registered by teachers at a school; 4) ratio of daily attendance in PD courses registered by teachers at a school to total possible days attendance; and 5) ratio of teachers registered for PD to total teachers at a school.

There were six performance variables used as dependent variables in the analyses. These included: 1) average school-level ISAT math scale score in 2005; 2) average percentage of students meeting or exceeding (%ME) state math standards in 2005; 3) average math scale score in 2006; 4) average percentage of students meeting or exceeding (%ME) state math standards in 2006; 5) change in the percentage of math students meeting or exceeding (%ME) state math standards between 2005 and 2006; and 6) Change in the average ISAT math score between 2005 and 2006.

The analyses began by comparing schools with and without an assigned CMS. This was done in two ways. First, the analysis looked at cross-school comparisons by examining mean differences in grades 3-5 between schools with and without an assigned CMS. Second, using the same population of schools it looked specifically at individual student-level changes in performance for the population of students who had taken both the 2005 ISAT math exam and the 2006 ISAT math exam.

The study included only students who were 3rd graders in 2005. (K-2 students and students in grades 4 - 5 in 2005 fell out of the analysis because they lacked two successive years (2004-2005 and 2005-2006) of ISAT data). cite   The analysis was limited to students in grades 3-5 because the central question of this study concerned the effects of EDM implementation and EDM was only implemented in the lower grades.

Other schools used a different set of instructional material in grades 3-5 (Math Trailblazers, "MTB"). To insure valid comparisons between EDM-implementing and comparison schools, the analyses excluded schools implementing MTB. The analyses provided information on the effects of CMS assignment, but did not provide information on the effects of EDM implementation. Hence, after comparing schools with a CMS to those without, the research moved to a second stage of analyses.

In this stage, researchers examined EDM implementation results by comparing EDM schools with comparison schools. cite   The analysis provided empirical evidence of the effects of EDM on both "Theory of Action" and "Performance" variables, but nothing about whether the observed effects were a function of EDM implementation or the assignment of a CMS. Hence, a subsequent series of analyses was conducted to directly address the question of whether the observed findings in the CMS assignment and EDM implementation analyses were due to EDM implementation or CMS assignment. This subsequent analysis looked at four groups of schools:

The analysis was important because it attempted to separate the effects of the EDM implementation and CMS assignment variables.

Finally, the study looked at the effects of a teacher trained under the "EDM Leadership Program." These analyses examined whether the training (or more specifically, the presence of an EDM leadership trainee at a school) caused any additional impact.

First, the studies tested whether the EDM Leadership Program had any direct influence on the findings of the CMS and EDM analyses. Second, they tested whether an EDM leadership trainee at the school had any indirect effect by interacting with EDM implementation or if CMS presence made a difference.

Eight groups were defined based on the permutations of the three dichotomous variables of interest. (EDM Implementing "Yes/No", CMS assigned "Yes/No"; and EDM Leader Trainee "Yes/No"). This process yielded eight school groupings:

Group EDM Implementing CMS Assigned EDM Leader Trainee
1 No No No
2 No Yes No
3 Yes No No
4 Yes Yes No
5 Yes No Yes
6 Yes Yes Yes
7 No Yes Yes
8 No No Yes

Two groups (Groups #7 and #8) were null sets. Each of these involved “non-implementing” schools. Analysts aggregated and merged data to create a school-level file. The data analysis used mean breakdowns and the introduction of independent and control variables, e.g., EDM and CMS status, to test for differences among the six viable alternative groupings.cite

The citywide math specialist studies were limited in a number of ways. First, the analyses focused exclusively on EDM implementation issues rather than all of CMSI. Second, the analyses are based on a single year's change and, as noted earlier, the annual summaries and %ME analyses have shown this to be a questionable practice. Third, while restricting the analyses to EDM and breaking the analyses down by grade provided control over program and grade level effects, the analyses failed to control for demographic differences among EDM-implementing and comparison schools.

Previous Page Next Page

The reason this analysis was only possible for this subgroup was tied to the ISAT testing process used by the Illinois State Board of Education (ISBE) prior to the 2005-2006 school year. Prior to the 2005-2006 school year, ISBE tested students in mathematics statewide in the 3rd, 5th, and 8th grades only. In 2005-2006, ISBE began standardized testing in all grades 3-8. However, only those students who were in the third grade in 2004-2005 and 4th grade in 2005-2006 met the criteria for inclusion. 8th grade students were excluded because, by definition, they were not in grades K-5. Students in grades K-2 were excluded because they do not take the ISAT. Thus, there were no 2004-2005 test data available for these students. Students that were in 5th grade in 2004-2005 were excluded, because they were in 6th grade in 2005-2006. Hence, they were not covered by the K-5 program during the second test year.
To eliminate any data contamination resulting from the inclusion of data from schools implementing the alternative K-5 program (MTB) in the "non-EDM Implementing" group, schools implementing MTB were again excluded from the analyses.
Group #7 included non-implementing schools with an assigned CMS and an EDM Leadership Trainee.

Group #8 involved schools that were not implementing EDM did not have a CMS assigned but had an EDM Leadership Trainee. Group #7 presented a "null set" because the group was defined to have both an assigned CMS and an EDM Leadership Trainee but were not implementing EDM. There were no such schools. By definition schools with an assigned CMS and/or an EDM Leadership Trainee were involved in implementing an OMS sponsored curricula.

Group #8 schools presented a "null set" because these schools by definition had EDM Leadership Trainees without an assigned CMS and were not implementing EDM. There were no such schools, since only schools implementing EDM had teachers involved in EDM Leadership training. However for logical completeness we list both of these null groups in the list of possible groups.