Center for the Study of Systemic Reform
   in Milwaukee Public Schools

 


Report on the Information System

INFORMATION SYSTEM DESIGN IN SUPPORT OF REFORM EFFORTS

Christopher A. Thorn
Director of Technical Services, WCER
University of Wisconsin-Madison

Abstract

Description of the information system. The Milwaukee Public Schools (MPS) are in the midst of a major data system upgrade that should have a profound impact on the ability of district administrators, principals, and teachers to access and use data to make informed decisions that bear on systemic reform. The first elements of the new School Management System are about to go into beta testing. The system is an outgrowth of a multi-year strategic planning process that outlined the need for a distributed data system to complement the District's decentralized decision-making structure. The development of the system is behind schedule at this point and is hampering the District’s progress towards achieving a model data-based decision-making structure.

Strengths and weaknesses. MPS has a long history of maintaining a rich database at the individual student-level. The data base was enhanced by the previous requirements of the Federal Title I program that required annual testing of students receiving federal funds. This data-rich environment has changed as the District has shifted its assessment strategy away from standardized tests to locally developed performance and proficiency testing. While such systems might better reflect the teaching and learning taking place in Milwaukee, they present a difficult problem for the analyst attempting to assess the validity and reliability of the District’s accountability system. A major factor working in the District’s favor is its strategic planning process. The strategic plan goes beyond identifying specific technical goals. It links data users with the questions they need to answer. This work provides the overall design architecture for the data warehouse system’s development. The data-driven seminars being run by the Office of Research and Assessment are a vital professional development tool for providing school-level administrators with the skills to interpret the complex performance data the system is being designed to provide.

Recommendations and future research. The information system is far too complicated to assess in the short time period this study has been underway. However, there are a number of problems facing the District and its technical and research organizations. Our research should provide support for both data consumers and for those designing the data warehouse. Our team that includes specialists in assessment, data systems, accountability, professional development, special populations, and alignment. We would like to work with the MPS Technical Services and Research and Assessment offices to improve the District’s ongoing data-driven decision-making efforts. We are also prepared to provide expert assistance to the data warehouse development team. The data team will both work to support the other project teams and coordinate their data needs, as well as work directly with District staff to assist in the development of new windows into the MPS data warehouse.

I. Introduction 

This section of the report addresses what we see as the major data needs of a systemic reform effort and the systems that would need to be in place to meet those needs. The discussion can be divided into the following four areas: 1) What kind of analyses should be done? 2) What variables do we need to measure (or count) in order to do these kinds of analyses? 3) How would one manage the processing and analysis of MPS data to be able to do the necessary analyses on these variables; and 4) Where can we make a difference in the immediate future? The primary activity of the data analysis team during the first six months of the project has been to understand both the data structures currently in place and the major uses of system-level data for compliance and analysis.

Methodology of the Study

This phase of the study is based on the extensive documentation of the data system and some of its school-assessment-focused outputs provided to us by MPS staff. We have examined strategic planning documents for an integrated district-wide information system, the request for a proposal for a new School Management System, the student record database code book, and a number of school report card and other school performance publications.

II. What kind of analyses should be done and for what purpose?

The Milwaukee Public Schools District serves approximately 106,000 students. The data- reporting requirements for the district are both numerous and complex. The current system was built incrementally in response to changing internal data needs and the changing information requirements of external actors--primarily state and federal agencies. The result of this type of system development is a central database that is designed to fulfill a number of specific management and reporting needs. What such a system lacks is the more sophisticated integration needed to provide a multidimensional window into the complex processes and outcomes that one must investigate in order to make informed judgements about systemic reform efforts. For example, individual schools often need disaggregated data for activities such as gauging local program effectiveness among particular subgroups. Centralized reporting systems also tend to lack the flexibility to be useful at lower levels in the organization.

On the analytical front, the requirements of program evaluation and performance assessment far exceed the requirements of most federal and state reporting requirements-- the traditional driving force behind educational data systems. One particular focus of this effort is to move toward a value-added approach to program evaluation. A value-added analytical framework works best if it is supplied with relatively frequent assessment measures and individual student-level data.

III. What variables do we need to measure (or count) in order to do these kinds of analyses?

This is a broad issue. The literature on student assessment and program evaluation suggests a number of possible additions to the traditional core data elements of student demographics, attendance, program eligibility, and assessment. These areas include frequent student assessments with item-level scores retained, fine-grained student/family demographics, and detailed school and classroom characteristics. These three areas are based on our understanding of the dynamics of student performance. These recommendations are based on the author's experience and the work of the National Forum on Education Statistics.1

Assessment

The focus of any school performance system has to be on the instruments used to assess student learning. We have identified three major areas of concern within the MPS system. The first is that there are large gaps in testing. Other than the state-mandated third-grade reading test, there is no district-wide testing until the fourth grade and not much subject area testing until the sixth grade. Such testing is particularly critical in the effort to identify and diagnose student performance deficiencies, especially as the deadline for the introduction of high-stakes testing in the fourth grade approaches. Without repeated, reliable diagnostic tests in prior grades, it is virtually impossible for MPS to know whether it is preparing students adequately for the high-stakes examination.

A second area of concern involves links between assessment instruments. Without careful psychometric analysis, one cannot be sure that the various standardized tests, performance assessments, and proficiency examinations are reliable or valid. Assessment data need to be retained at the item level and the use of any assessment instrument should be backed up with a psychometric analysis of linkages to other instruments and instructional standards.

Title I requirements, which mandated extensive annual testing of aid recipients, provided Milwaukee with a long history of frequent testing across the District. This goes against the state-level tendency to rely on aggregate data. The MPS information system keeps extensive records at the individual student level. Our concern is that, as the reform efforts focus on local proficiency and portfolio assessment measures, the District not abandon externally administered, norm-referenced tests. District plans call for extensive monitoring of student performance. It is important that these performance instruments be closely aligned to District curricula and that they be validated internally through psychometric analysis and studies of intercoder reliability. Linked, externally validated evaluation instruments would also provide a crosscheck on the consistency of the district's assessment system.

Finally, several structural issues inherent in the demographics of the District allow a significant number of students to be exempted from the accountability system. The information system failed to pick up the test scores of several categories of students--highly mobile students; students with exceptional educational needs; and limited-English-proficient students who are released from some testing requirements. This is not a process of neglect, but it does focus the discussion of student outcomes solely on general education students. The District has an obligation to educate the non-tested students and has programs in place to do so. There needs to be a formal reporting mechanism to acknowledge school and district performance with these populations.

Student/Household Demographics

The existing MPS core information system contains primary student demographic information-- ethnicity, free and reduced lunch status (as a proxy for income), and exceptional education status. Other records exist for identifying each student's exceptional need(s), but they are stored in other areas of the information system and one must understand a complex linkage structure to interpret the information. Existing research on student performance tells us that there are other important factors for assessing the impact of schooling on student outcomes. The following variables are some examples of data that would improve our understanding of each student's educational context:

Parental/guardian education level
Distinctions of ethnicity (Hispanic and Asian students of differing nationalities are lumped into these two categories)
Integrated presentation of exceptional education status
Student mobility history

School/Classroom Characteristics

School and classroom characteristics are another important data type. These factors can be lumped together in that they represent a complex combination of monetary and personnel resources. If we consider decisions about class size, professional development, curricular initiatives, investment in technology or other capital equipment, and local data storage and analysis systems, there are a number of important factors or interventions that could have a significant impact on student outcomes.

Attendance area (neighborhood versus city-wide programs)
Professional development activities
Level of teacher training
Teacher experience
Investment in administrative and instructional technology
Class size
Special curricular focus (Montessori, language immersion, etc.)

This is something of a laundry list of data elements that could improve the District's ability to analyze performance data. We acknowledge the difficulty and high cost of collecting some of these data. The logistical difficulties of data collection and the cost of testing will continue to provide a barrier to high-level analysis of school performance data.

IV. How would one manage the processing and analysis of MPS data to be able to do the necessary analyses on these variables?

The practical implications of a robust systemic analysis framework are daunting. MPS officials are not unaware of this problem. In its Technology Strategic Plan2, the planning committee outlined specific data needs for teachers and school administrators that are a direct result of district decentralization. The following excerpt from the report's Executive Summary outlines the technology needs of the three levels of the organization:

Classroom Management in a Decentralized Organization
Instructional time can be increased by reducing teacher time spent on classroom management tasks like attendance and grade record keeping. A single point of data entry (the teacher) should distribute that information across the school. New technology can then make available that data and integrate all other data relevant to a particular student to assist staff with decision making and the provision of services.
MPS has taken steps toward redesigning the student information database maintained at the district level. In addition, a site-based transaction-oriented database system is required. The two databases together can exchange relevant student information to provide better support.
School Management in a Decentralized Organization
Decentralization has imposed staggering new responsibilities on school management personnel at the same time that the complexity of client needs has increased. School-based technology will help address these challenges.
MPS Accountability in a Decentralized Organization
Systemic integration of reporting data at both the school and district level is required to tie together school educational plans, school accountability measure reports, district monitoring reports (MPS report card), state reports, and federal reports.3
This portion of the strategic plan was then used to develop a Request for Proposal (RFP) for a new School Management System to enhance and extend the existing information system's capabilities. The two major themes of the Technology Strategic Plan and the RFP are "providing data to drive local and district decision-making" and assuring that the system "support school innovation by providing a tool that allows schools to implement their own initiatives and educational models."4 These two goals imply an information system that is both a decision-support system that is linked to district and local goals, as well as one that has the capacity to design new data acquisition and reporting capabilities linked to local needs. Either of these goals by itself would be difficult to achieve. Achieving them simultaneously will take both innovative programming and high-level training for the intended users in the schools.  

The RFP lays out global system requirements that address some of the major shortcomings of the existing system. These requirements include an integrated security model, an import/export facility, and a user-friendly query and reporting capability. These prerequisites are important features that the present system lacks. The document goes on to elaborate on the current situation and projected needs in all of the major data subsystems. The distance between the existing capabilities of the system and the projected end points are sometimes quite significant. One of the most positive elements of the RFP is the theme of data-based and data-driven decision-making. One of the major considerations to be faced when one is designing a database is to understand the questions that will be asked of the data. Much of the RFP is focused on improving the timely collection and reporting of student data: attendance, guidance interventions, discipline, grades, etc. It is also clear that the new School Management System will be used to evaluate individuals, programs, and processes. The needs of the district require universal access--the ability to access a particular set of records from any location--and real-time longitudinal elements that track changes as they occur over time.  

The shift from a centralized data storage and reporting system to a responsive, pervasive decision-supported system will be a difficult challenge. The client/server topology recommended in the Technology Strategic Plan and required in the RFP provides a division of processing power and data accessibility that reflects the needs of actors at different levels in the system. The proposed system incorporates the two primary models of client/server system design. First, individual school administrators and teachers will be able to query the central data repository from their own computers. Second, the data queried can be downloaded to a local computer for further manipulation, or for combining with local data. The central data store might also supply "what if" data-sets that allow for the development of contingency planning based on changes in important systemic variables.5 Most importantly, the system being developed will allow people at a distance from the central office to become sophisticated consumers of student and system process data.

V. What progress have we made?

We now have a much clearer understanding of what assessments are available for analysis in each grade at the district level. We have also begun to build a longitudinal database of student demographic, mobility, and performance data that will help us to make intelligent suggestions in order to aid in the design of the emerging School Management System in MPS. In cooperation with other members of the research team, we are working to provide links to segments of the student population that are (for various legitimate reasons) not part of traditional assessment system.

Technical challenges at the District level have reinforced our impression of the level of difficulty one can expect to encounter when building such complex data-sets. Reporting problems and lack of physical data system infrastructure (disk space and extensive backup libraries) within MPS have made gaining access to the necessary data a time consuming and drawn out process. The new data system being designed now should reduce these problems, but problems continue to hamper the district's ability to perform many analytical tasks.

VI. What steps should be taken to address issues beyond our control or resources?

There are other, wider issues involved in changing school and administrative culture (and analytical capabilities) to make active use of the complex information system being proposed. One aspect of building decision-support systems that is often overlooked is the human element. Managers operating under the current system are socialized to make decisions in certain ways--often with a limited amount of data upon which to base their decisions. An integrated information system will not change this culture by itself.

Driver and Svensson (1996) suggest that there are five basic decision styles (or habits) that differ across two dimensions. The first is the amount of information used to make a decision. The second dimension is the number of possible solutions considered when looking at a particular situation. Driver and Svensson's research suggests individual decision styles affect the way in which people access and use information. Their findings also support the argument that managers who received training on being sensitive to their own data needs and use were more likely to consider more options when making decisions. They were also more likely to be aware of the decision-making style of a manager in a related unit. What this means for MPS is that the technical training required in the RFP will only provide part of the answer for moving towards a data-based decision-making model. The ongoing Data-driven Decision Making seminars being sponsored by the MPS Research and Assessment Office may prove to be the catalyst for this sort of shift in models of decision making.

There is no doubt that the improvements called for in the School Management System’s RFP will provide more accurate and detailed historical data for the MPS system. The plan calls for data portability, timely entry and access, user-friendly query and reporting features, and flexibility. There are also elements that will increase the availability of data to all relevant constituencies (students, teachers, administrators, parents, etc.) through different types of online technology. However, the analytical demands made on the individuals receiving the data may be more than they are prepared to cope with.

School and student performance is a difficult topic to study under the best of circumstances. In a large urban district with a highly mobile low-income population, the complexity of any useful statistical model will not be minor. Differences in attendance rates or test scores, for example, contain a high level of variance and often contain systemic measurement errors related to local school conditions, such as self-selection into particular programs or mobility related to a parent's migrant-worker status (for additional information, see Rubenfeld & Newstrom, 1994). A system that provides more data will very likely not produce the desired reform results. Without a vigorous professional development plan that includes strong methodological and analytical training for teachers and administrators, the new School Management System will be of limited analytical usefulness.

VII. Where can we make a difference in the immediate future?

  1. One obvious area where the capability to create new data files for analysis will be increasingly important is in the direct support of the data-driven decision making seminars it will provide. We would like to provide additional analytical support to the MPS Research and Assessment Office and, perhaps, for a cross-section of schools participating in the seminars. This activity aligns almost exactly with our own interests and capabilities. All of our project staff has attended various seminar sessions this fall. We feel we can provide both feedback and training resources.
  2. It is in the area of decision support design that our data analysis team could be most helpful, to both the MPS Research and Assessment Office and the MPS Technical Services department. We have expressed our willingness to join the development team for the data warehouse interface as this effort comes back online. We hope to work with MPS staff at both the District and school level to make the best use of existing data and to add data elements that provide new insights into the dynamics of student performance. We also plan to help data consumers focus on creating new data structures that best support their educational and administrative goals.
  3. We would like to continue to develop our own longitudinal database as a vehicle for testing different types of analytical approaches. As part of this effort, we could work with Technical Services staff to improve the District's ability to export data for analytical uses. The ability to assemble the historical data available in the MPS data system into a longitudinal student-outcome database is vital if one is to analyze the impact of program changes on school and student performance.
  4. We would like to work with MPS officials to discuss building a number of decision-making models based on the present understanding of student achievement and school performance. This would, ideally, be part of the new data warehouse interface. These models would provide baseline data on student and program outcomes and would establish an analytical framework against which individual school or classroom models could be studied.6
These models could include value-added approaches to school and student performance. They might also provide comparison of programs across different student populations. The design effort would be focused on providing models for evaluating the outcomes that were identified as important by system constituencies at the different organizational levels.

These models each would exhibit three characteristics. One, they would be based on robust historical data. Two, the models in question would receive a rigorous methodological review. And three, the design would be informed by collective understanding of the important factors affecting student and school performance.  

Footnotes

1See National Center for Education Statistics, 1997.

2"The Impact of  a Client/Server Architecture on Decision Support Systems," by M. Whitman and H. Carr, 1994, The Executive's Journal, 10, p. 12. http://whscdp.whs.edu/tsp/plan/

3Ibid. pp. 8-9.

4MPS. RFP-239, p. 0-3.

5For more on this see, for example, M. Whitman & H. Carr. Information Strategy, Winter 1994, Vol. 10 Issue 2, p. 12.

6See, for example, Poole (1995), or Alavi (1994), for example of decision support systems used to enhance group decision making.

References

Alavi, M. (1994). Computer-mediated collaborative learning: An empirical evaluation. MIS Quarterly, June, Vol. 18 (2), 159.

Driver, M., & Svensson, K. (1996). A human-information-processing approach to strategic change. International Studies of Management and Organization. Spring, Vol. 26 (1), 41.

Milwaukee Public Schools System. (1996). Milwaukee Public Schools' Technology Strategic Plan, December 11 (Rev. 02/01/97) http://whscdp.whs.edu/tsp/plan/

______. (1997). RFP-239: School Management System, October.

National Center for Education Statistics. (1997). Basic data elements for elementary and secondary education information systems. Washington, DC: GPO, U.S. Department of Education.

Poole, M. S. (1995). Decision development in computer-assisted decision making. Human Communication Research, 22 (1), 90.

Rubenfeld, S., & Newstrom, J. (1994). Caveat emptor: Avoiding pitfalls in data-based decision making. Review of Business, 16 (2), 20.

Whitman, M., & Carr, H. (1994). The impact of a client/server architecture on decision support systems. The Executive’s Journal, 10, 2.

 

 

 

Information Staff Reports Links