Center for the Study of Systemic Reform
   in Milwaukee Public Schools

 


Alignment RESEARCH
 

Norman L. Webb
Senior Research Scientist
Co-Director, Center for the Study of Systemic Reform in Milwaukee Public Schools,
WCER, University of Wisconsin-Madison  

Abstract  

Our main focus during the first six months of 1999 in the Milwaukee Public Schools (MPS) study has been to study the alignment among important MPS components. We did an analysis of the alignment of the MPS Standards and Grade Level Expectations with the Wisconsin Model Academic Standards for four content areas. In general, we found that the MPS Standards and Grade Level Expectations were comparable in content and depth-of-knowledge in language arts, mathematics, and social studies. We had more difficulty in determining the relation of the two documents in science. In the science content area, there appeared to be less agreement in content coverage common to both, (with only about 60% agreement as compared to 85%-100% agreement in other content areas), and a higher percentage of the MPS standards were of a lower depth-of-knowledge level. MPS curriculum specialists and teachers are currently reformatting the standards in a more consumable form for teachers’ use. Further analysis of the standards documents and how these documents are used by teachers and principals is needed to fully understand how these are linked with other system components and how they contribute to high achievement among students.  

I. Introduction 

One premise of systemic reform is that the major components of an education system must work together to guide the process of helping students achieve higher levels of understanding (Smith & O’Day, 1991, Zucker et al., 1998).  Educators recognize that if system components are not aligned, the system will be fragmented, will send mixed messages, and will be less effective (CPRE, 1991; Newmann, 1993). For example, the systemic initiatives program of the National Science Foundation (NSF) is directed toward states, districts, and regions setting ambitious goals for student learning that are based upon a coherent policy system. The Improving America's Schools Act explicated how assessments are to relate to standards: ". . . such assessments (high quality, yearly student assessments) shall . . . be aligned with the State's challenging content and student performance standards and provide coherent information about student attainment of such standards . . ." (U.S. Congress, 1994, p. 8). Similarly, the U.S. Department of Education's explanation of the Goals 2000: Educate America Act and the Elementary and Secondary Education Act (which includes Title I) indicated alignment of curriculum, instruction, professional development, and assessments as a key performance indicator for states, districts, and schools striving to meet challenging standards. 

Milwaukee Public Schools has important pieces in place for an aligned system that is capable of concentrating its efforts toward improved student achievement. The district has written content standards and grade level expectations; developed assessment systems and student proficiencies for grade 8 to grade 9 promotion (in 1999-2000) and for graduation from high school (in 2003-2004); and established a standards-based assessment system, with related intervention programs and professional development. The district also operates with school-based decision-making that allows principals and their staffs some autonomy. The district is aligned to the degree that all of these components are working toward the same ends. The main goal of the research for this study is to determine whether all of the components are aligned. In the pursuit of this goal, we seek to answer specific questions about the district: 

·        Is there a clear vision among teachers, administrators, and students of what students are to know and do?

·        Do teachers and schools provide students with the opportunity to learn what they are to know and do?

·        Do MPS assessments give accurate information on students’ progress and attainment of what they are to know and do? (See also, Robert Meyer’s report on analyses of MPS assessment data, pp. 29-34). 

II.  Progress to Date  

In the first six months of the grant, May through November, 1998, our major effort was to become acquainted with the district. We identified the district’s assessment system; tracked the historical developments in mathematics, science, language arts, and social studies; identified the expectations for student learning; and began identifying the range of professional development experiences available to teachers (Clune & Webb, 1998). After the Board of School Directors adopted the MPS standards in November 1998, we concentrated our efforts on two activities. We sought to verify that the adopted standards were aligned with the Wisconsin Model of Academic Standards, one criterion of the validity of the MPS standards. We tried to understand how useful the standards documents are in providing district staff the information they need to guide student learning. Over the period from January through May 1999, we conducted a content analysis of the alignment between the state curriculum standards in language arts, mathematics, science and social studies with the MPS curriculum standards. 

We engaged in other activities as requested by district staff and used these activities as opportunities to understand MPS reform efforts. We reviewed the proposed mathematics performance assessment activities prior to administration in March and sent Kathy Swope, Office of Research and Assessment, our comments on February 19, 1999. We sent her our comments on the proposed science performance assessments on March 29, 1999. On February 24, 1999, the alignment research team met with Jocklyn Smith and curriculum specialists to present our approach to thinking about and studying alignment in the district and to seek their cooperation. By March 4, 1999, we had completed interviewing five curriculum specialists, including Carmen Baxter, science; Karen Villcock, mathematics; Sharon Durkta, social studies; Sandra Dickerson, language arts; and, Frankie Johnson, reading.  

In April, we identified eight elementary and middle school sites for gathering more data on the use of the standards in schools, the alignment between the instructional experiences students received in schools and the standards, and how the degree of alignment relates to student learning. A team of three to five researchers conducted one- and two-day visits at three schools by the end of the school year. Other school visits are planned for the fall.  

As a new initiative, related to our objective of developing district capacity for more effective use of information as a basis for instructional decisions, we prepared and submitted a proposal to the Joyce Foundation. The proposed work, in cooperation with the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) at the University of California, Los Angeles, is to implement the Quality School Portfolio (QSP) in MPS middle schools and study how teachers and principals are able to use this tool to make more effective decisions. QSP is a school-based electronic information system designed to help school staff make more effective use of data.  

Development of a Standards-Based System at MPS

Over the past six years, the district has been on a trajectory leading towards a standards-based system and increased student achievement. Such large-scale reform takes time to reach coherence among all of the district's components. There is evidence in all of the four content areas—language arts, mathematics, science, and social studies—that progress has been made in aligning the important components of standards, curriculum, assessment, and professional development. Important steps towards a standards-based system include:  

·        Adoption of K-12 Teaching and Learning Goals (about 1994);

·        Development of grade-level expectations in mathematics and science (from 1994);

·        Mandated district graduation requirement and middle school proficiencies (1996);

·        Implementation of writing and mathematics performance assessments;

·        Milwaukee Urban Systemic Initiative (initiated 1996-97);

·        MPS Standards and Grade Level Expectations (1998);

·        District-wide adoption of mathematics curriculum, grades 6-8 (1998-99); and,

·        Professional development on new mathematics curriculum.

·        Mandated high school graduation requirements.

In November 1998, the Milwaukee Board of School Directors approved the Milwaukee Public Schools K-12 Academic Standards and Grade Level Expectations for language arts, mathematics, science, and social studies. Responding to demands of the state and building on work that had already been done, the MPS Division of Curriculum and Instruction developed these content standards. The standards and grade level expectations in each content area were developed under the leadership of the content area curriculum specialist and with the help of teacher committees. The work in each content area took a different approach, based in part on what was already in existence. As a consequence, the format of standards and grade-level expectations among the four content areas are different.  

In language arts, the curriculum specialist generally took the Wisconsin Model of Academic Standards and then nested grade-level expectations under these by detailing what students had to learn at each grade level in order to attain the standards at grades 4, 8, and 12. Grade-level expectations were already available for mathematics and science. The curriculum specialists in mathematics and science worked with teachers to specify general standards of what students were to study. Most of these standards stated only the grade range, the grade-level expectations were appended to the standards without describing the relation between the standards and the grade-level expectations. Over the 1998-99 school year, teachers and the curriculum specialists worked on reformatting the standards in a column format that would explicate the relationship between standards, grade level expectations, performance standards, and assessments. Three committees of teachers for social studies, one for each grade range, built standards for their grade range based on the Wisconsin standards. The high school group adopted the state standards organized by major topic (world geography, history, etc.). The elementary group essentially took the state standards and wrote grade-level expectations for each intervening grade. The middle school group wrote their own standards and then modified them to include a local transition from elementary to high school and a similar format.  

III. Findings

If standards are to drive the system and contribute to improving student achievement, it is essential that the standards represent important content, that they are succinct, that they can be understood by diverse populations, and that they provide direction for curriculum, professional development, assessments, and classroom practices. During the spring semester, we conducted an analysis of the MPS standards in the four content areas, using Wisconsin’s Model Academic Standards as a model of high quality standards. Our preliminary analysis indicates that there is strong alignment within language arts, mathematics, and social studies in the content topics students are expected to have knowledge about and in the depth at which students are expected to know these topics. From 85% to 100% of the topics in the Wisconsin Model Academic Standards were at least partially incorporated in the MPS Standards and Grade Level Expectations, and from 79% to 94% of the topics from the MPS standards were at least partially incorporated in the Wisconsin standards. There were a number of reasons for the difference in content coverage, including:  

1.      MPS Standards and Grade Level Expectations are more detailed statements than those in the state standards;

2.      Some differences exist in the grade level at which students are expected to acquire specific knowledge; and

3.      A vagueness in language makes it difficult to judge the relationship between documents. 

We found that the alignment in science was more difficult to decipher because of the difference in structure between the MPS Standards and Grade Level Expectations and of the Wisconsin standards. Our preliminary analysis found it difficult to establish close matches in statements of student expectations for nearly half of the MPS grade level expectations with the state standards for grades 4, 8, and 12. For example, under Standard C, Life and Environmental Science, the MPS grade level expectation for grade 8 states, “Describe how some diseases are the result of internal failure of the body systems” (C.8.4).[1] Under the same topic for grade 12, an MPS grade level expectation states, “Identify and describe the structures and functions of the human body system” (C.12.24). There was not an explicit match with either of these statements in the Wisconsin standards. The Wisconsin standards under Life and Environmental Science used more general terms that possibly could include these expectations; but with the information provided, it was very difficult to make a clear decision. For example, a grade 8 Wisconsin standard stated, “Differentiate between single-celled and multiple-celled organisms (humans) through investigations, comparing the cell functions of specialized cells for each type of organism” (F.8.3). A grade 12 Wisconsin standard stated, “Evaluate the normal structures and the general and special functions of cells in single-celled and multiple-celled organisms” (F.12.1).

We also considered how demanding the MPS standards are in relation to the Wisconsin Model Academic Standards. We used four levels of depth-of-knowledge to compare what students were expected to do for MPS and the state:   

            Level 1            Recall of information

            Level 2            Skills and conceptual knowledge

            Level 3            Strategic thinking

                        Level 4             Extended thinking 

In language arts and social studies, the depth of knowledge students were expected to attain by the MPS Standards and Grade Level Expectations and the Wisconsin Model Academic Standards were nearly identical. In general, the percentage of expectations with a higher depth-of-knowledge level increased in the higher grades. In language arts, about 65% of the grade 4, 53% of the grade 8, and about 27% of the grade 12 expectations and standards for both MPS and Wisconsin were judged to be a Level 2 (Skill/Concept). The percentage of language arts standards at Level 3 (Strategic Thinking) increased by grade from about 22% for grade 4 and 32% for grade 8 to 47% for grade 12.   

In social studies, for grades 4, 8, and 12, about 20 % of the standards were rated at Level 1 (Recall). However, for grades 4 and 8, the highest percentage of standards were rated as Level 2, from 50 to 70%. As could be expected, for grade 12 the highest percentage of standards were rated as level 3, about 55% for both MPS and the state.

MPS standards and the state standards had a similar pattern in the depth-of-knowledge levels for mathematics. Both sets of standards have a high degree of Level 2 standards. Level 2 standards dominate grade 4, comprising about 75% of the standards. For grades 4 and 12, about 50% of the standards were rated as Level 2 and about 40% of the standards were rated as Level 3 for both MPS and the state.

A noticeably higher percentage of the MPS standards for science was rated as Level 1 compared to the state standards, from 26% to 32% for MPS compared to 2 to 18% for the state. For grades 8 and 12, the state had a higher percentage of standards rated as Level 3 than did MPS, about 20% compared to about 30%. This indicates that a greater number of the state standards required students to evaluate, predict, or analyze, whereas more of the MPS standards required students to describe, identify, observe, apply, or illustrate.   

IV. Next Steps: Policy Recommendations and Future Research             

  1. The standards adopted in November, 1998, need to be put in a common format that will help teachers and others understand the difference between what students are to know and to do and how their understanding of the content area should be structured. MPS is already working on this task.
  2. More detailed analysis is needed on the relationship of the district standards and grade-level expectations with the full range of assessments used in the district.
  3. It is evident now that more rigorous procedures are needed for district-developed assessments to assure that the different forms are equivalent, valid, and reliable. A list of development procedures were sent to Kathy Swope, Office of Research and Assessment, on April 28, 1999. The proficiencies, measured in part by district-developed assessments, appear to exert a significant influence on instruction. This provides an additional reason to assure that these instruments are of high quality and measure the full depth-of-knowledge in the intended areas as specified in the content standards.
  4. We will continue to study the alignment within MPS on two fronts. In the summer, June 30 and July 1, we will conduct an alignment analysis of the relationship between the MPS Standards and Grade Level Expectations with the assessments in mathematics and science. This will provide more detailed information on both the standards and the assessments.
  5. By the end of August, we will complete the analysis of data from the three schools we visited in May. These data will give us more information about what is directing teachers’ instruction in their classrooms and how this is shaped by the assessments, district mandates, and the standards. We will visit additional schools in the fall semester to extend this work and to help verify our findings about the degree to which MPS standards, assessments, and classroom practices are aligned.
  6. Our previous and future work on alignment will help identify specific actions that can be taken to assure that:  

·        teachers, administrators, and students of have a common vision of what students are to know and do;

·        students are provided the opportunity to learn what they are to know and do;

·        assessments provide accurate information on students’ progress and attainment of what they are to know and do;

·        assessment activities to better reflect what is in the content standards; and

·        knowledge in certain content areas is being adequately assessed.  

We also will identify the articulation (vertical alignment) of standards and assessments across the grades, which is so important if students' learning experiences are to effectively accumulate over time.

7.      We will continue to gather data through reviewing documents and interviewing people. Also, we will continue to forge a relationship with the MPS Office of Research and Assessment staff to contribute to developing more valid assessments. We will prepare a written report in August on our analysis of the three schools. This will be shared with the staff at each school and with district staff. Other reports will be prepared as work is completed.

References 

Clune, W. H., & Webb, N. L. (1999).  Technical reports on Milwaukee Public Schools policy, information systems, and instructional alignment.  Madison: University of Wisconsin, Wisconsin Center for Education Research.  

Consortium for Policy Research in Education. (1991). Putting the pieces together: Systemic school reform (CPRE Policy Briefs). New Brunswick: Rutgers, The State University of New Jersey, Eagleton Institute of Politics.  

Newmann, F. M. (1993). Beyond common sense in educational restructuring: The issues of content and linkage. Educational Researcher, 22 (2), 4-13, 22.  

Smith, M. S., & O’Day, J. (1991).  Systemic school reform.  In S. H. Fuhrman & B. Malen (Eds.), The politics of curriculum and testing. Politics of Education association Yearbook. (1990, pp. 233-267).    

U.S. Congress, House of Representatives. (1994, September 28). Improving America’s Schools Act. Conference report to accompany H.R. 6 Report 103-761. Washington, DC: U.S. Government Printing Office.  

Zucker, A. A., Shields, P. M., Adelman, N. E., Corcoran, T. B., & Goertz, M. E. (1998) The National Science Foundation’s Statewide Systemic Initiatives Program: Evaluation Report. Menlo Park, CA: SRI International.


Footnote

[1]  Identifiers designate a specific MPS or Wisconsin standard: e.g., this reference is to MPS Life and Environmental Sciences Standard C, Grade 8, Grade Level Expectation 4.   

 
Information Staff Reports Links