2003 Standard Setting for the Wisconsin Knowledge and Concepts Examination (WKCE)

Wisconsin Knowledge and Concepts Examination (WKCE) was developed to assess student performance in core academic areas in accordance with the Wisconsin Model Academic Standards. For each grade-and-content area, the WKCE is designed to identify accomplishment at four different performance levels: Minimal Performance, Basic, Proficient, and Advanced. The staff of the Office of Educational Accountability (OEA) at the Department of Public Instruction (DPI) worked with its test vendor, CTB/McGraw-Hill, to establish proficiency-level cut scores for the Reading, Language Arts, Mathematics, Science, and Social Studies sections of the WKCE for grades 4, 8, and 10. The establishment of these performance levels was done through the use of the Bookmark Standard Setting Procedure (BSSP) in February 2003.

BSSP is a proprietary method of setting standards that was developed by CTB/McGraw-Hill for the specific purpose of simultaneously accommodating selected-response and constructed-response test formats, simplifying the cognitive complexity required of standard-setting judges and connecting test content with performance-level descriptors (Mitzel et al, 2001). The general framework for utilizing a bookmark standard-setting method provides participants with an ordered-item booklet. In this booklet, the items are ordered from the easiest to the most difficult, based on obtained difficulty scores from previous administrations of the items. The ordered-item booklet has one item per page, with the first item being the easiest, and the last item being the most-difficult. Each participant is instructed to place a bookmark at the location in the item booklet where the participant believes that a proficient student needs to master the knowledge, skills, and abilities contained in the items before the bookmark. Items preceding the participant's bookmark reflect content that all students at the given performance level are expected to know and to be able to perform successfully with a probability of at least 0.67. Conversely, these students are expected to perform successfully on the items behind the bookmark with a probability of less than 0.67. Three rounds of making bookmarks are typically conducted before results are summarized and before a final recommendation is made.

Participants

From February 9 through 12, 2003, a workshop was held in Madison, Wisconsin, with Wisconsin educators charged with the purpose of identifying the requirements for successful performance on the WKCE. Two hundred and forty panelists from 84 school districts attended the workshop; they included teachers from grades 3, 4, 7, 8, 9, and 10 as well as teachers of students with learning disabilities, of bilingual students and those with Title-I needs; administrators; curriculum experts; school-board members; guidance counselors; assessment coordinators; other curricular-area teachers, e.g., art and music; business leaders; parents, and other citizens from across the state.

Participants were nominated by professional organizations, by themselves, and by other referral sources. Each nominee was asked to provide qualifications and experiences. The Department of Public Instruction selected the most-qualified panelists who also represented various areas of Wisconsin, who brought a rich variety of experience, and who collectively provided a balance of gender, ethnicity, and other factors.

Approximately 20 panelists participated in each grade-and-content area. Within each grade-and-content area, the participants were divided into three small groups. This was done to facilitate discussion between and among participants and to make the process more manageable for DPI and CTB/McGraw-Hill staff.

Method

The Bookmark Standard-Setting Procedure that was conducted for the WKCE consists of several steps that were designed to produce a recommendation for where the four proficiency level standards should be set. Each of these steps was essential in order to produce an informed decision that adheres to generally-recognized industry standards for validity and reliability.

After the initial orientation, the first step of the process was for participants to complete the operational form of the test. This was done to give participants a more-thorough understanding of the test content. Once this was done, participants were given copies of the examination items in order of difficulty (from easiest to most difficult) with one item per page. This Ordered-Item Booklet (OIB) was then analyzed, and participants were asked to identify what each item measured as well as why each item was more difficult than the preceding items. Following this, participants were provided training on how to place bookmarks. The bookmark placement required participants to move individually through the ordered-item booklet and to place a bookmark where the level of item difficulty for each category was surpassed. One bookmark was placed for each of the required cut points between Minimal Performance and Basic, between Basic and Proficient, and between Proficient and Advanced. Items preceding the mark reflected content that all students at the given performance level should have been expected to know and should have been able to do, defined as a likelihood of at least two-thirds.

At the beginning of the second stage of the BSSP, participants were provided with aggregate and disaggregate impact data based on the round-one placement of the bookmarks. Participants were then provided with the opportunity to discuss within their small groups the effect of placing the bookmarks at the selected locations and to identify the range of items between the lowest and highest bookmark for each proficiency level. Once this group activity was completed, participants were instructed to conduct the bookmark placement activity again individually.

The third stage of the BSSP was conducted in a fashion similar to that of the second stage. Participants were again provided with aggregate and disaggregate impact data based upon their recommendations and were given the opportunity to discuss their individual results within the small group. Participants were given the opportunity to place their individual bookmarks for a third and final time.

Final Recommendation

The final recommendations from the BSSP participants for each grade-and-content level were based on the final median-bookmark placements. These recommendations were then forwarded to the Wisconsin Technical Advisory Committee for their review and recommendations. This was then forwarded to the State Superintendent of Public Instruction, who adopted the recommendations of the standard-setting group and of the technical advisory committee.

References

Lewis, D. M., Mitzel, H. C., & Green, D. R. (1996). Standard setting: A bookmark approach. In D. R. Green (Chair), IRT-based standard-setting procedures utilizing behavioral anchoring. Symposium conducted at the Council of Chief State School Officers National Conference on Large-scale Assessment, Phoenix, AZ.

Lin, Jie. (2004). The bookmark standard setting procedure: Strengths and weaknesses. The Centre for Research in Applied Measurement and Evaluation, University of Alberta: Edmonton, Alberta, Canada.

Mitzel, H. C., Lewis, D. M., Patz, R. J., & Green, D. R. (2001). The bookmark procedure: psychological perspectives. In G. J. Cizek (Ed.), Setting performance standards: concepts, methods, and perspectives (pp. 249-281). Mahwah, NJ: Erlbaum.

For questions about this information, contact oeamail@dpi.wi.gov

Last updated on 2/17/2009 9:21:26 AM