Using a Continuous Progress Indicator to Review the Progress of Schools

May 1999

The expectation that there will be ". . . continuous and substantial yearly improvement in each. . . school sufficient to attain the goal of all children . . .meeting the state’s proficient and advanced levels of performance."
20 U.S.C.A. § 6311(b)(2)(B)

Introduction
For a number of years (1993) state law has required that the department (DPI) identify schools in need of improvement based on performance data from the state’s knowledge and concepts exams. Federal law (Improving America’s Schools Act, 1994) similarly requires the state and local districts to annually review the progress of schools to determine if they are enabling students to meet the state’s proficiency standards. Both state and federal law emphasize progress of schools and students.

Over the past several years the benchmark used to identify schools as in need of improvement has been based on an average of students’ performances statewide, i.e., 90% of a statewide average. This was a figure calculated based on statewide performances which changed each year. Using comparisons to the average or using point in time determinations are not the most desirable measures of progress. In the spring of 1998 there were 10 policy forums held around the state to seek input into our state policy for determining progress. We asked, ‘What percentage of students should be expected to achieve at the proficient level or above?’ and ‘What amount of progress should be considered adequate?’

Section II of this paper describes the concerns raised at the spring 1998 policy forums. Section III describes key features of a new indicator of progress, the "continuous progress indicator," which was developed to address these concerns and will be used beginning this year. Section IV provides an explanation and examples of how the "continuous progress indicator" will be computed. And Section V explains how the "continuous progress indicator" will be used, along with the current school performance compared to the statewide average, to review the progress of schools.
Concerns Raised at the Adequate Yearly Progress Forums in May, 1998
As a result of the forums we learned several over-riding concerns from participants:
- Schools should be credited for overall improvement in the number of students in the Proficient and Advanced categories and/or for moving students from the Not Tested or Minimal category into Basic or above while not decreasing the number in the top two categories.
- Provision should be made to address potential swings in performance that can occur in schools with small numbers of students at each grade level.
- Schools may find it more difficult to make progress as they approach higher achievement levels.
- Annual progress requirements may be difficult to meet over time; and meeting these requirements might be more difficult for states like Wisconsin where student achievement and proficiency standards are already high.
- Previously proposed progress requirements may be unrealistic for many schools. Little justification exists for expecting a certain percent improvement. More data over time is needed on progress of schools under Wisconsin’s new accountability system to validate progress measures employed.
These concerns led us to hold off on the adequate progress approach which was under consideration. In the fall of 1998, Superintendent Benson advised the districts of his decision to stay with "90%" of the statewide average as the benchmark for the 1998-99 school year; holding off on a progress measure until the 2000-2001 school year. Reflecting on this "status quo" decision however led to growing concern over the appropriateness of this standard: As with student test results expressed in percentile terms, this standard simply compares school performance to an average of unknown quality and fails to adequately recognize progress. The progress approach proposed last spring held the promise of allowing school districts to choose to adopt continuous improvement, measuring their progress against themselves over time.

We have therefore been exploring different models of continuous improvement for schools. The model which is outlined below is the result of our further study with review by our external Technical Review Panel of assessment experts, and two additional focus groups composed of a variety of educator groups from across the state. These review groups have found the model to be relatively easy to understand and reasonable as a continuous progress indicator (CPI) for districts’ schools. This model not only addresses the concerns identified in the spring 1998 forums, it will allow a school district to use the traditional "90%" of the statewide average or to switch to the continuous progress indicator (CPI) as their improvement indicator for a particular school beginning this spring of 1999 and into the future.
A New Way to Review Progress Using a Continuous Progress Indicator
At a glance, the following features summarize the elements of the continuous progress indicator:
- The Continuous Progress Indicator (CPI) would focus on improvement/movement to the Basic level and to the Proficient or Advanced level. Earlier models focused solely on a percentage of students at or above Proficient.
- Each academic year, a CPI would be calculated based on the four most recent years of test results. Earlier models used the two most recent years. Using four years of data addresses concerns, especially concerns of smaller schools, about variation in the characteristics of successive groups of students at the given grade level.
- A required CPI would be established for each school based on the school’s baseline achievement level. Greater progress would be expected in schools where fewer students are Proficient or Advanced.
- Schools would not be identified for improvement unless the CPI falls below the required level for two consecutive years.
- There would be no required CPI for schools that have above 80% of their students scoring at the Proficient level or above. Schools would not be identified for improvement as long as 80% of the students score at or above the proficient level.
- The required CPI would be uniform across subjects and grade but vary by school achievement level in the baseline year.
- The required CPI will be re-evaluated as more data on progress of schools under Wisconsin's new accountability system become available.
- The use of a CPI in the review of school performance was discussed with 2 focus groups and the DPI Technical Review Panel consisting of national assessment experts. All three groups thought this approach has merit.

Determining the Continuous Progress Indicator

Use of the CPI in reviewing the progress of schools is based on an expectation that the percentage of students reaching the Proficient level or above in the WKCE subject areas will continue to increase until all, or nearly all, students have attained that goal. The CPI is intended to serve as the primary basis for reviewing the effectiveness of a school in helping each successive group of students in grade 4, 8, or 10 reach the Proficient level or above. There is no prescribed time frame for attaining the goal.

The CPI is different from previously proposed measures of progress in that this Indicator recognizes decreases in the percentages of students in the "Not Tested" and "Minimal Performance" categories in addition to increases in the percentages at or above the Proficient level. This change makes the CPI more sensitive to improvements in all schools, thus emphasizing the continuous progress notion for both low- and high-scoring schools. An example of how to calculate the CPI for 1998-99 is provided in Table 1.

Table 1
An Example for 1998-99

Calculating the CPI for Science
Any High School — Grade 10

Percents of Students Enrolled for a Full Academic Year

Year of Testing	Not Tested on WKCE	Minimal Performance	Basic	Proficient	Advanced	% Not Tested or Minimal	% Proficient or Above
1998-99	7%	10%	31%	34%	18%	17%	52%
1997-98	10%	15%	21%	35%	19%	25%	54%*
*Baseline % At or Above Proficient. See Table 3.					Improvement:	+8%	-2%

CPI for 1998-99: 6

Note that Improvement is positive if there is a reduction in the % Not Tested or Minimal and/or an increase in the % Proficient or Above.

Another difference between the CPI and other proposed measures is that four years rather than two years of proficiency scores would be used each year in measuring progress. Since 1997-98 was the first year of proficiency reporting, sufficient data to implement this feature of the CPI will not be available until after 2000-01 testing.

The use of four years rather than two years of scores to calculate the CPI tends to reduce the effect of error on the Indicator. This is particularly important in measuring progress of very small schools. Using four years of data would also address concerns about variation in the characteristics of successive groups of students at the given grade level.

For 1999-2000, three years of proficiency scores would be used to calculate the CPI because only three years would be available. Results for 1999-2000 would be compared to the average of the results for 1997-98 and 1998-99.

Beginning with test results for 2000-01, the CPI would be calculated based on the four most recent years of test scores. The average of the two most recent years would be compared to the average of the two years preceding that period. For 2000-01, this means that the average of 1997-98 and 1998-99 would be compared to the average of 1999-2000 and 2000-01. See Table 2.

Table 2
An Example for 2000-01

Calculating the CPI for Mathematics
Any High School — Grade 10

Percent of Students Enrolled for a Full Academic Year

Year of Testing	Not Tested on WSAS	Minimal Performance	% Not Tested or Minimal	Two-Year Average Not Tested or Minimal	Basic	Proficient	Advanced	% Proficient or Above	Two-Year Average Proficient or Above
2000-01	2%	19%	21%	25%	35%	29%	15%	44%	44%
1999-2000	4%	24%	28%		28%	34%	10%	44%
1998-99	7%	25%	32%	36%	27%	32%	9%	41%	38%*
1997-98	10%	29%	39%		26%	25%	10%	35%
	Improvement in % Not Tested or Minimal:			11%		Improvement in % Proficient or Above:			6%

*Baseline % At or Above Proficient. See Table 3.

CPI for 2000-01: 17

In the school described in Table 2, if only two years of proficiency data were used, then the CPI for 2000-01 would be 7 rather than 17. To understand this calculation, look at the column "% not tested or minimal." From 1999-2000 (28%) to the 2000-2001 (21%) school year there was a 7% improvement.

If the CPI is as large or larger than the Required CPI below, then the school has met the improvement requirement for that subject and grade. Schools at lower achievement levels would be expected to make more progress than other schools. For 1998-1999, the Baseline % At or Above Proficient would be the % Proficient or Above for 1997-98. For 1999-2000 and 2000-01, the Baseline % At or Above Proficient would be the average of 1997-98 and 1998-99. For 2001-02, the Baseline % At or Above Proficient would be the average of 1998-99 and 1999-2000 and so on. See Table 3.

Table 3
Improvement Requirements Per Academic Year

Baseline % At or Above Proficient:	00%-20%	21%-40%	41%-60%	61%-80%	80%
Required CPI:	4	3	2	1	Stay above this level

In the school described by Table 1, the Baseline % At or Above Proficient was 54% in science. Based on Table 3, the Required CPI is 2. If the 1998-99 CPI for the school is 6, as shown, then the school exceeded the Required CPI in science.

In the school described by Table 2, the Baseline % At or Above Proficient was 38% in mathematics. This is the average percent Proficient or Above for 1997-98 and 1998-99. Based on Table 3, the Required CPI is 3. The 2000-01 CPI for the school is 17, as shown, then the school exceeded the Required CPI in mathematics. For 2001-02, the Baseline % At or Above Proficient would be the average percent Proficient or Above for 1998-99 and 1999-2000, i.e., 43%.

No specific amount of progress would be prescribed for schools that exceed 80% at or above the Proficient level as long as 80% of the students score at or above the Proficient level. Continued progress of these schools, to the goal of all or nearly all students attaining the Proficient level, will be considered as DPI clarifies its policy regarding the identification of Distinguished Schools.

Integrating the Use of 90% of the State Average with the CPI

Through school year 2000-01, annual review of school performance will be based on the following:

Percent scoring at the Proficient level or above in each subject is at least 90% of the 1997-98 statewide percent in that subject. This is called the "90% rule."

OR
Meet the Required CPI for each subject

Table 4
Criteria for the Review of School Performance Over Time

Year of Testing:	1997-98	1998-99	1999-2000	2000-01
Required Achievement Level or CPI:	Meet 90% rule. No CPI.	Meet 90% rule OR Meet the Required CPI based on school year 1997-98 data compared to 1998-99 data.	Meet 90% rule OR Meet the Required CPI based on the average of 1997-98 and 1998-99 data compared to 1999-2000 data.	Meet 90% rule OR Meet the Required CPI based on the average of 1997-98 and 1998-99 data compared to the average of 1999-2000 and 2000-01 data.

These criteria would apply to reviews for reading, mathematics, science, and social studies. The criteria for language arts will be one year behind due to changes in the way language arts will be considered for accountability purposes beginning with the results of 1998-99 testing. The 90% rule for language arts will be based on 1998-99 test results.

Beginning with test results for school year 2001-02, the use of the 90% rule would be discontinued in all subject areas. The CPI would be fully implemented; the previous four years of data would be used for annual review purposes.

Note that only scores of students enrolled for the full academic year prior to testing are considered. With February testing this means three semesters or more at the same school. As has been true beginning with 1997-98 test results, schools would not be identified for improvement in a subject unless the Required Achievement Level or CPI indicated in Table 4 is below the necessary level for two consecutive years.

For questions about this information, contact oeamail@dpi.wi.gov

Last updated on 2/26/2008 10:42:50 AM