Mainstreaming Underserved Students in the K-16+ Continuum
to Achieve Academic Excellence

 

Manuel Gómez and Norma Dávila 
University of Puerto Rico

 

Abstract

The University of Puerto Rico’s Resource Center for Science and Engineering has pioneered a pipeline model that brings together major Science and Mathematics reform at the K-12 and 13-16+ levels. A virtual organization, the RCSE establishes strategic alliances to catalyze and sustain the reform that primarily targets students from low-income families who are typically underserved by the system. The empirical theory that serves as the foundation for the reform is introduced as well as the systemic models and methods utilized. Evidence of success in achieving reform goals is presented using the assessment, attribution, and accountability feedback cycle (A3). An example of how this reform can be disseminated, transferred, and adapted to other settings such as New York City is presented. 


Overview and Context 

This paper is an analysis of the major elements of the K-16+ systemic education reform strategy used in Puerto Rico to mainstream underserved students (low-income students) into the educational continuum. The paper also discusses how this systemic approach can be transferred and adapted to inner city schools and to universities in the mainland.

The issue of equity in Puerto Rico is cast in socioeconomic terms.  The middle class and affluent parents send their children to private K-12 schools. In contrast, low income parents, particularly those below the federally defined poverty level, send their children to public schools under a single school district that is administered by the Puerto Rico Department of Education (PRDE) and fully funded by Puerto Rico’s state government (see Figure 1).  The majority of the population in Puerto Rico (66%) is below the federally defined poverty level and most of the parents in this group do not have more than a high school education. This large fraction of the population hinders the economic development of the Island that, by any standards, is essentially a high-tech manufacturing economy with 40% of its GDP is due to manufacturing; thus, the importance of Science and Mathematics education to the Island’s well-being.

 

The number of students in Puerto Rico’s K-12 system is 850,000, of which 23% (200,000) go to private schools and the majority 77% (650,000) is enrolled in PRDE public schools.  These numbers highlight the fact that educational reform on the Island is dealing with one of the largest educational systems in the United States. The size and complexity of the challenge requires a systemic approach to reform if significant improvement in academic excellence and academic performance of students is going to take place.

Students from private schools systematically outperform those from  public schools in all standardized tests and, in particular, in the College Entrance Examination Board’s (CEEB) university admission test which is the Spanish equivalent of the SAT in the mainland.  Private school students on the average perform 70 points better than public school students in the College Board’s Mathematics Reasoning test, a difference that is comparable to the one that exists in SAT’s Mathematics aptitude test scores when the majority population is compared with Hispanics. As a result of this difference, a disproportionately large number of private school students are accepted to college. For example, at the University of Puerto Rico (UPR), Río Piedras Campus—the oldest and most prestigious university on the Island—68% of the admitted students come from private schools which is a threefold overrepresentation of this socioeconomic group.

This brief description of the educational and socioeconomic reality of Puerto Rico provides enough evidence that a valid correspondence can be drawn between public and private schools in Puerto Rico and inner city and suburban schools in the mainland.  Discussions of the transferability and value of the pipeline model for systemic educational reform developed in Puerto Rico to underserved populations on the mainland will be based on this correspondence.

During the past twenty years, the UPR—through its Resource Center for Science and Engineering (RCSE)—has been pioneering systemic reform in the K-16+ continuum following a pipeline model (Culotta & Gibbons, 1992). Following this model, the RCSE identifies major bottlenecks or stumbling blocks to a student’s progress through the educational pipeline and develops a systemic approach to ensure that a larger number of students are mainstreamed and complete their college education.  The paper will discuss the systemic model and methods utilized to achieve reform at the K-12 and 13-16+ levels and evidence will be provided on their success in achieving their goals. 

At  the  K-12  level,  during the past seven years, the Puerto Rico Statewide Systemic Initiative (PR-SSI)—an NSF sponsored initiative—has undertaken a systemic reform of the teaching and learning of Science and Mathematics in Puerto Rico’s public schools.  Using the school as the unit of change, a transformation of their teaching/learning institutional culture has been attained where all students, use standards-based rigorous curricula and are exposed to effective teaching/learning methods that have improved their academic performance.  Through the establishment of 28 school-based Dissemination Centers, a scaling-up strategy has been implemented where 50% of all public schools in Puerto Rico have been or are in the process of being reformed.  Assessment and evaluation have been used as powerful tools to drive the reform and impact the accountability system and the allocation of systemic resources.  As a result of these strategies, the graduating seniors of fully reformed public schools have improved their performance in the CEEB’s (SAT equivalent) test to the point that they outperformed middle class private school students.

In order to complete the educational reform cycle, an NSF sponsored Collaborative for Excellence in Teacher Preparation (CETP) has been put in place to reform Teacher Preparation at six institutions of higher education where 80% of the future public school teachers in Puerto Rico receive their education.  The future teachers are being taught Science, Mathematics, and Pedagogy, following the same educational principles and precepts that have made the PR-SSI successful.

At the 13-16+ level, a project to reform institutions of higher education has been underway for eight years under the NSF sponsored LSAMP (Louis Stokes Alliance for Minority Participation).  The mission of this project is to increase the effectiveness and efficiency of the educational pipeline to increase the number of underrepresented students who complete a bachelors degree in Science, Mathematics, Engineering, or  Technology (SMET) and go on to graduate school in these fields.  By reforming gatekeeper introductory courses and developing special strategies for students at risk, a significant increase in number of Bachelors degrees in SMET fields conferred by UPR has been achieved—from 1,200 in 1991 to 2,100 in 1998—at a time when enrollment in these fields at UPR was going down.  The quality of these graduates is attested by the fact that, in the UPR-Río Piedras Campus, one out of every ten students who finishes a B.Sc. goes on to obtain a Ph.D. in SMET.

To transform complex educational systems, a catalytic agent is needed to orchestrate the reform, forge the necessary alliances, analyze the system, and identify the weak or missing connections among the system’s elements. Because of the magnitude and complexity of the task, a virtual organization that is not part of the existing administrative structure and that is capable of thinking and acting across systemic boundaries is needed.  The RCSE has played this role in designing and implementing the K-16+ reform model to improve the participation and mainstream underserved students in the educational pipeline. A discussion of the essential elements that such a virtual organization must incorporate will be discussed.

Finally, the issues of dissemination, transferability, and adaption of successful reform strategies will be analyzed.  An example of a dissemination model in action will be presented using the Federal Department of Education sponsored project entitled: “Puerto Rico/New York City Educational Linkages Demonstration Project”, in which twelve of New York City’s educational system schools located in the South Bronx are being reformed using the whole school approach and innovative teaching methods pioneered by the PR-SSI.

 

A K-16+ Seamless Continuum To Mainstream Underserved Students: The Resource Center for Science and Engineering's Pipelline Model

Rationale of the Pipeline Model

Many efforts have been undertaken to increase the participation of underrepresented groups in the educational K-16+ continuum to increase their participation in the SMET workforce (Oakes, 1990).  Yet, progress in achieving this goal has been spotty at best.  A powerful analogy to help understand this problem has been the concept of an educational pipeline (Culotta & Gibbons, 1992) where the loss of students from the educational continuum is interpreted as leaks in the pipeline due to inefficiencies in the educational process or to accumulated content and cognitive deficiencies in the students’ education.  Most efforts to fix the educational system have been too narrow in scope, too specific, or applied inconsistently, with little or no effort going to make the educational system more effective.

The RCSE, created in 1980 with NSF funding, has the mission “to nurture a new generation of Puerto Rican scientists and engineers”.  Since its inception, it embraced the concept of the educational pipeline as a powerful metaphor to analyze the K-16+ continuum using systemic thinking as it applies to complex social systems.  The business sector has been cognizant of the importance of conceptualizing large multinational corporations as complex social systems in reengineering their organization and management in order to achieve higher effectiveness and retain their competitiveness in the global economy (Ackoff, 1999).  

The RCSE analyzed the whole continuum and concentrated efforts in understanding the interaction and feedback between subsystems (i.e.: K-12, the undergraduate institutions, the community, and the political establishment) and then developed strategies to address the bottlenecks and barriers to educational progress.  It became obvious from the beginning of the reform effort that the common practice of using intervention programs to increase educational participation of underserved groups only tweaks the system without changing the fundamental impediments to progress which are systemic in nature.  If permanent and long-lasting change is to be achieved, an institutional cultural transformation is required at the different levels of the educational continuum. Thus, the need to establish strategic alliances between the different levels of the system and to use catalytic strategies to modify their institutional behavior. 

 

 

Taking seriously the systemic approach, the RCSE designed a long-range plan to reform the educational continuum and increase its capacity to move students, who are at risk of dropping out, effectively through the pipeline.  By identifying multiple sources of external funding and serving as an agent of change, the RCSE promoted coherence and synergy between the different  projects to produce a seamless K-16+ continuum.  See Figure 2 for a list of all the programs that are articulated by the RCSE at the different levels of the educational pipeline. 

Orchestrating this complex portfolio of different projects and harnessing them into a coherent systemic reform strategy is far from simple and requires an organization that can serve as the catalytic agent of reform and the broker of needed strategic alliances.  Furthermore, complex social systems, as is the case with the educational systems, are highly non-linear, dynamic, and reactive. These qualities require that the implementation of the reform sequences activities and develops a very good sense of timing on when to introduce them if coherent long-term results are to be achieved.  Even more important, the available reform resources are never commensurate with the resources of the system; thus, lasting reform can only be achieved if the considerable resources of the system can be reconfigured to sustain the reform.

Like with an orchestra, coherence and harmony can only be achieved in a systemic reform effort if there is a conductor.  This role has been played by the RCSE that has acted as a virtual organization.  The concept of the virtual organization was developed by the RCSE in parallel to the development of the same concept in the corporate world.  In industry, the virtual organization was created to bring together different corporations into a strategic alliance to achieve a common goal that, because of its nature, lies outside the individual corporation’s goals.  Its salient characteristic is that it is delocalized in space and time: human resources needed to achieve the goal are usually retained in their corporate setting and most of the communication is done using modern technology that permits delocalization in time. 

Similarly, the RCSE acts as virtual organization by helping to forge strategic alliances of the different sectors or institutions of the educational system, harnessing multisector resources to achieve the reform goal, and using modern communication technologies to accomplish the task.  The RCSE is a strategic partnership between institutions of higher education, the PRDE, the Government, and the private sector.  By being outside of the regular educational system’s organizational structures, the RCSE can serve as a catalytic agent for reform, nurture alliances, and keep a systemwide view of the goal and the implementation plan (see Figure 3 for the nature of the interaction between the RCSE and the different elements of the system).

 

 

Because of its comprehensive nature and the scope of its charter, the RCSE interprets the education pipeline to encompass from the kindergarten level all the way to Ph.D. and postdoctoral levels. The RCSE carries the reform further to include the development of the UPR into a Research 1 institution developing R&D and contributing to the economic development of the Island.  At the K-12 level, the PR-SSI is the central project; at the 13-16, the LSAMP project and CETP Teacher Preparation programs lead the way, while, at the 16+ level, EPSCoR (R&D) and EPSCoT (Technology Transfer) projects are leading the reform. Since the purpose of this paper is to discuss the K-16 part of the continuum, only the K-12 and undergraduate reform efforts will be discussed in the following sections.

Harnessing the System's Resources to Implement and Institutionalize Successful Reform 

Once a strategic alliance has been wrought, reform goals have been conceptualized, reform resources have been obtained, and a virtual organization to spearhead the reform has been put in place, the reform cycle must be implemented.  Most systemic reform efforts have failed because the reform fails to implement key elements of the cycle that are illustrated in Figure 4.  In most systemic reform efforts, the strategic plan and the pilot stage of implementation usually go forward with varying degrees of success, but, when the scaling-up stage is reached, most reform efforts fail because they are not able to harness the system’s resources.  Also missing from most reform efforts is the design and implementation of systemwide evaluation that will document improved student outcomes and the modification of the system’s process of accountability and allocation of resources to meet the needs of the reformed schools or universities.  The absence of this key element breaks the reform cycle and, thus, the full transformation of the system.

 

Three elements must be coherently developed and articulated in order to complete the reform cycle’s Assessment, Attribution, and systemwide Accountability (A3), (see Figure 5).  Of these three elements, the least understood is attribution, defined in the following manner: the capacity to establish a “quasicausal” and persuasive relationship between the implementation of reform strategies and the improvement of student learning.  If the system’s authorities that control resources and have the power to: 1) evaluate the system; 2) provide resources for scaling-up; and 3) control the accountability and allocation of resources processes are not persuaded that the pilot tested reform efforts are not superior to standard practice, the authorities will not give way to the reform.  The role of the reformer is to take the positive results of the assessment of pilot projects and harness them into a coherent set of facts and information that can make a persuasive case for attributing students’ improved learning and educational progress to reform efforts. Otherwise, wide dissemination and scaling-up of the reform will not take place.  This paper will discuss, under K-12 and 13-16 reform, how the linking of Assessment, Attribution, and Accountability (A3) has been used to drive the reform and produce permanent changes in the system.

Summary:

The main features of the pipeline model of reform discussed here—1) identifying the major weak points in the pipeline; 2) systemic thinking in the design and implementation stages; 3) development of strategic alliances; 4) strategic application of limited reform resources; and 5) developing a virtual organization to orchestrate the reform and harness the system’s resources to transform the system are critical to any systemic reform project if it is going to be successful. The fact that the RCSE took the larger challenge of tackling the whole pipeline, K-16+ should not dissuade anyone from applying these reform principles to smaller systems such as: K-12 or even subsets of the K-12 continuum.  But, no matter what is the size and scale of the subsystem that is selected, the key players with the decision-making power to make changes and harness the system’s resources for the reform must be brought into the strategic alliance.

 

The K-12 Science and Mathematics Reform (PR-SSI)

The PR-SSI Alliance 

After experimenting with intervention programs for more than 10 years--through the RCSE--it became apparent that, unless the public educational system could be transformed into a mode of operation where all students could learn — irrespective of their socioeconomic level—the underserved students in the public educational system would not be mainstreamed into the educational pipeline.  This was the rationale for designing a systemic initiative to reform the PRDE’s public schools so that all students could learn Science and Mathematics. 

A strategic alliance was forged between the major institutions of higher education, under the leadership of the RCSE, the PRDE, and the Community. The Alliance conceptualized and developed a strategic ten-year reform plan and secured catalytic reform funds from the National Science Foundation’s  Statewide Systemic Initiative Program.  Funds were committed by PRDE and UPR on a ratio of 2 to 1 with respect to NSF funding. These resources provided the necessary funds to pilot test the “Whole School Approach” in the first cohort of seven participating schools. The reform then was scaled-up using these schools as Dissemination Centers. 

The RCSE —an alliance of the major institutions of higher education  administered from the President’s Office of the University of Puerto Rico—served as the virtual organization that orchestrated the K-12 reform and harnessed the alliance’s members into an effective operational coalition.   The PRDE exercised its leadership to establish rigorous nationally accepted standards for the teaching of Science and Mathematics, pioneered policies to decentralize school academic and administrative management by creating the Community School concept, and established systemwide assessment for student performance in Science and Mathematics.  As the reform has progressed through the scaling-up stage, it has gradually harnessed more system resources to support new schools. At present, 50% of all public K-12 schools have or are being developed into reformed schools.

The School as the Unit of Change and the Scaling-up Strategy

Attempts to reform large complex systems in toto fail because the reformer is unable to handle the complex dynamics of such systems and because reform resources are not commensurate with the reform challenge.  But more important, complex social systems obey the principle of stasis; they resist any attempt to change them and, if change is made, they tend to return to their previous state when the reform efforts end.  Thus, a catalytic approach is called for.  A unit of change needs to be defined at a scale small enough where reform strategies can be pioneered, pilot tested, and assessment can be performed to make a valid case for attribution.  The unit of change can be seen as the smallest unit in the system that still shows systemic characteristics and that can be used as the basic unit to disseminate and scale-up the reform.  For example, the PR-SSI selected the school as the unit of change while Massachusetts-SSI selected the school district as the unit of change.

Scaling-up requires that the reform husbands limited human and infrastructure resources.  It also requires a strategy to develop an increasing cadre of qualified reformers who are capable of disseminating the reform to an ever increasing number of units of change (schools or districts); they also need to learn how to harness existing system resources to contribute to the scaling-up effort.  Furthermore, effective reform requires an institutional cultural transformation that changes the knowledge base and the individual and collective behavior of the community that constitutes the unit of change, namely, teachers, principals, and parents. The preferred way to achieve institutional cultural transformation is through carefully designed, coherently implemented, and focused professional development for teachers and educational managers in the basic unit of change.

To scale-up the reform, the PR-SSI developed the concept of Dissemination Centers. Schools whose staff had been imbued in the reform’s principles and became adept at their application and whose teachers had shown leadership qualities were selected and prepared by the PR-SSI to serve as Regional Dissemination Centers. These Dissemination Centers are geographically dispersed and serve as foci of reform and as professional development centers in their regions.

Transforming the Teaching/Learning Culture of the School: The Whole-School Appoach

Many of the systemic reform projects attempted to reach the whole system through massive widespread professional development (Corcoran, Shields, & Zucker, 1998; Mervis, 1998) because they defined the whole system as the unit of change. The lack of focus and of a well conceived scaling-up strategy of these projects ignored the principle of stasis that all large and complex social systems obey. The reform effort became just a perturbation on the system and the little systemic transformation that the reform achieved was dissipated when the stasis of the system brought it back to its previous state (Corcoran, Shields, & Zucker, 1998).

An educational system that is not achieving its intended goal of educating students and increasing their academic performance to meet rigorous Science and Mathematics standards has, by definition, some degree of dysfunctionality, hence the need for educational reform.  To correct for dysfunctionality, a systemic approach is necessary to transform the institutional culture. Taking seriously the concept of unit of change, the PR-SSI undertook the task of transforming the teaching/learning culture of the school.

In order to be systemic in approach, the whole school was analyzed to determine what was impeding that all students could learn and could achieve challenging educational standards and improved academic performance. The following factors were identified: a) lack of rigorous Science and Mathematics standards against which to measure progress; b) inadequate standards-based Science and Mathematics curriculum; c) absence of a coherent professional development program tailored to the specific content and teaching methodology needs of the teachers; d) lack of empowerment of teachers to take effective control of their classroom’s teaching/learning process; e) absence of a true community of learners in the school; f) lack of access to technology and effective curricular and laboratory equipment and materials; and g) absence of management methods and styles that would nurture and support experimentation and change. 

During the pilot stage, the PR-SSI experimented with various school-based reform strategies and approaches. It soon became evident that a linear sequential approach would not be systemic, but, furthermore, that it would not achieve the needed transformation of the school’s teaching/learning culture. A non-linear coherent approach was called for in which all seven weaknesses would be addressed simultaneously and the inherent synergy and interdependence of these seven elements would be exploited to enhance the reform effort. The “Whole School Approach” to systemic reform was conceptualized.

The “Whole School Approach” requires that all the Science and Mathematics teachers in a school and their principal recognize and accept the need for reform and that they all agree to be prepared through summer and school year professional development activities. To implement this approach, the PR-SSI announced, disseminated, and gave workshops on the goals of the reform and on the positive results that had been obtained in the initial seven pilot schools. The PR-SSI afterwards opened a competition for schools that wished to join the reform process by submitting proposals where they analyze their need for reform, demonstrate their commitment to  reform, and all Science and Mathematics teachers and the principal agree to attend the summer and school year professional development activities.

The engine that drives the “Whole School Approach” and has permitted the scaling-up from the original seven pilot schools to 800 schools (50% of the 1600 public schools under the PRDE), has been the A3 strategy;   it has provided the persuasive evidence that reform works when the “Whole School Approach” is followed. As a result, at present, more schools wish to enter the reform than the resources of the PRDE and PR-SSI can handle. 

The requirement that all Science and Mathematics teachers in a school agree to participate in the reform is an essential element of the “Whole School Approach”. This strategy permits the school, with the appropriate training for teachers and principals, to become a true community of learners that will carry on the reform beyond the initial professional development stage. As the case study of Japan under the TIMSS (United States Department of Education, 1998) illustrates, one of the strongest characteristics of Japanese schools is that teachers act as a community of learners; teachers cooperate and learn from each other in improving their lessons and teaching methods.

The development of communities of learners and teacher empowerment have been key elements in transforming the teaching/learning culture of the schools to attune them to the reform. They are also the best antidote to systemic stasis since they provide internal self-correcting mechanisms to sustain and further develop the reform effort. The professional development provided by the PR-SSI includes specific workshops on the social dynamics and methods needed to develop a community of learners. The PR-SSI has also designed and implemented specific activities to develop the attitudes and skills needed to become an academically empowered teacher who can take full control of his/her classroom teaching/learning environment and actively participate in the school’s decision and policy-making process.

To implement the scaling-up of the reform, it was necessary to increase the cadre of qualified reformers who could provide the professional development and the academic and technical assistance that a school needs to go through the reform process. Initially, professional development and academic and technical assistance were provided by a cadre of university faculty and a carefully selected group of lead teachers. In order to increase the pool of reform resources, schools that had already been reformed with a high degree of success and that had shown leadership qualities were inducted into the reform efforts as Regional Dissemination Centers thus increasing the pool of active qualified reformers. At present, there are 28 Regional Dissemination Centers geographically distributed throughout Puerto Rico; these Centers provide professional development and academic and technical assistance to the schools in their regions. The PR-SSI staff closely monitors and assesses the Centers’ performance and uses strict quality control measures to ensure the continued high quality of the scaling-up process.

Principals play a key role in the ‘Whole School Approach” since they  control resources and have the means to demand accountability and exert  leadership to obtain additional resources for the schools. For this reason, as part of the reform effort, they attend workshops on leadership, the principles of systemic reform, the management of change, and how to steer a group of empowered teachers towards common goals. As a result, they have nurtured the reform in their schools and many have become reform leaders in their own right.

Ensuring that professional development is coherent and supports the reform effort requires a great degree of flexibility in its design in order to adapt it to the evolving needs of teachers and principals (Loucks-Horsley, Hewson, Love, & Stiles, 1998). All participating schools go through a core four-week Summer professional development experience to initiate them in the reform process, become fully cognizant of the Science and Mathematics standards, learn the pedagogical principles that drive the reform, learn to use the model curricula, and teach them how to change the teaching/learning culture of the school to meet reform goals.

In successive years, the PR-SSI’s professional development takes more the form of continued education. In this stage, the assessment results of student performance and of the teachers’ performance using assessment instruments developed by the PR-SSI staff are used to tailor professional development programs to the evolving needs and interests of the teachers. The school-based communities of learners also provide the means for the continued self-improvement of the teachers and the school beyond professional development activities.

 

A Course of Study Designed to Serve All Students: Mainstreaming the Underserved Student

The Challenge

Not surprisingly, underserved students –those with less than optimal access to quality education and a nurturing teaching/learning environment – are also academic underperformers. The educational system responds to their academic underperformance by a series of approaches that exacerbate the problem rather than solve it. These approaches include: tracking students into less challenging curricula; placing underperformers in remedial groups; and seggregating them in schools that lack resources and have lower standards of performance. The assessment and evaluation instruments that are used to separate high performers from low performers are also not diverse enough to allow the students to express what they have learned in an adequate manner or to take into consideration different learning styles and learning rates.

A plethora of intervention programs have been designed, funded, and implemented to move educationally underserved students through the educational pipeline with varying degrees of success. But, after more than thirty years of these intervention programs, the underrepresentation of minority population in Science and Technology still remains (Kimmelman, Kroeze, Schmidt, van der Ploeg, McNeely, & Tan, 1999). We posit that the reason for this underrepresentation has been that intervention projects, by their very essence, are non-systemic in nature while the problem of underachievement of the educationally underserved population is due to a dysfunctional system that is not designed to allow students from diverse backgrounds and multiple cognitive talents and learning styles to succeed. The PR-SSI, by design, was conceptualized and implemented to act on this hypothesis and to show that, if a systemic reform of the educational system is undertaken to provide opportunities for all students to learn, underserved students can at least equal the performance of mainstream students.

The essence of the PR-SSI reform was to transform the teaching/learning environment in the classroom and the school’s institutional culture to one that nurtures the multiple talents of a diverse school population and challenges them to learn using their full intellectual potential. The focus of the reform was to create schools that centered their effort on creating a positive and constructive learning environment rather than only to concentrate efforts in the teacher and his/her teaching capacities. The following key elements were incorporated into the reform: 1) establishing widely accepted rigorous Science and Mathematics standards to guide the teaching/learning process; 2) tranforming the teaching methodology to be student-centered; 3) promoting inquiry-based learning in which the student is encouraged to construct his/her own knowledge base; 4) promoting conceptual understanding by spiralling critical concepts from the concrete and simple to the abstract and complex; 5) providing multiple means of assessing the student’s knowledge and understanding; and 6) using the assessment to enhance rather than deter student learning.

As mentioned in the introduction, the PR-SSI became a natural experiment to test the hypothesis that schools attended by underserved students can perform as well as those serving more affluent students coming from families with parents from higher levels of schooling.  By concentrating its efforts in the reform of Puerto Rico’s public schools, the second largest educational district in the nation after New York City, the PR-SSI was poised to answer the following research question: Can schools serving low-income students perform equal to or better than schools where students come from more affluent and better educated families? The evidence that will answer this question will be discussed in a later section of this paper.

Curricular and Teaching/Learning Methods Used by the PR-SSI to Reach All Students

A curricular continuum from K-12 was designed that was aligned with the approved Science and Mathematics standards. The scope, sequence, and coordination of curricular content was designed grade by grade, the development of large themes and concepts was used to provide curricular cohesiveness within and between grades. The more fundamental concepts were spiralled throughout grades from the concrete and simple to the abstract and complex. The principle of “less is more”, less content and rote memorization and more depth of understanding, was used to determine the scope of the curriculum. Coordination of content between Science and Mathematics and within each discipline and between years was emphasized in the overall curricular design to ensure coherence and proper sequencing of the learning process.

Model curricular materials and activities were designed or adapted from exemplary national curricular materials to instantiate the standards and the grand curricular design. The following sound pedagogical principles grounded in educational and cognitive science research were used to guide the development and adaption of curricular materials: 1) the curricular materials and activities were designed to be student-centered with the teacher serving as a facilitator of the learning process; 2) inquiry and discovery hands-on activities were preferred; 3) materials encouraged students to construct their own knowledge and concepts following a constructivist approach; 4) learning experiences were designed to occur in the context of the students’ direct experience; and 5) most activities were designed to be used in the cooperative learning mode.

In the pilot stage of this reform, the PR-SSI found that teachers needed an intensive and focused professional development experience before they used the new curriculum and curricular materials in their classrooms. Professional development workshops were designed in which the teachers learned to use the materials by using them under conditions that modelled the students’ learning experience. The teachers were also provided with hands-on training in the use of innovative teaching methods like student-centered learning, use of the inquiry approach, and the management of cooperative groups. Finally, teachers learned to use a variety of authentic assessment tools to measure student learning. These tools included open-ended questions and scoring rubrics, performance-based assessments, portfolios, and reflexive diaries among others.

The macro curricular design, curricular materials, and activities were pilot tested in seven schools and a pre/post test assessment instrument was used to test their capacity to reach all students. The results of the assessment showed that all students could master the most basic concepts and thinking skills while still providing ample space for the best and most talented students to delve and explore deeper into the subject matter.

Semi-structured interviews of students and teachers as well as direct classroom assessments conducted by an experienced cadre of teachers and university faculty revealed that the improved learning by all students was due to the following principles and strategies: 1) student-centered learning; 2) heavy dependence on an inquiry and discovery approach to learning; 3) use of cooperative learning groups; and 4) use of multiple means of assessing student learning. This formed the base for the reform of schools using the “Whole School Approach”. The resulting impact on student learning for all students of the reform is discussed in the following section.

 

Harnessing Evidence of Success to Drive Systemic Reform and Retain Underserved Students in the Educational Pipeline

Assessment, Attribution, and Accountability (A3): The Drivers of Systemic Reform

To make systemic reform succeed, the barrier to change that the stasis of the large system presents, must be overcome, and the system’s resources have to be redirected to support and sustain the reform process. This requires that the base of the system – teachers, parents, and principals – be energized to accept and undergo the necessary changes needed to make the system more functional. At the same time, the top leadership – Secretaries of Education, Superintendents, and the Legislatures – must be persuaded to assign and redirect system resources and establish the necessary policies to support change. The role of the reform’s catalytic agent – the reform’s virtual organization – is to do the appropriate assessment of the change models and strategies, then harness the data into information and make a persuasive argument that the implementation of the models and strategies results in improved student learning and academic performance.  In other words, to establish a “quasi-causal” relationship between reform activity and improved student performance – attribution.

The PR-SSI followed this approach to drive the educational reform in Puerto Rico. By concentrating all of its reform efforts in the public school system which serves the majority of Puerto Rico’s underserved students. The PR-SSI had an ideal design to demonstrate that, with appropriate systemic educational reform, underserved students can significantly improve their academic performance and, in some cases, exceed the performance of affluent students attending schools with far superior resources. This section will provide the evidence that supports this statement.

By using the A3 approach, the PR-SSI was able to entice an increasing number of public schools to voluntarily join the reform process. In a period of three years, the number of schools in the PR-SSI reform has gone from 192 to [800] or 50% of all public schools in Puerto Rico (see Figure 6).  Actually, the rate limiting step to adding more schools has not been the number of schools willing to join, but rather the rate at which the cadre of reformers can be prepared and developed to sustain the reform.

 

 

The Puerto Rico Department of Education has put in place policies that support the reform effort such as: 1) all students are required to take Science and Mathematics every year from 7th. to 12th. grade; 2) the Community School Law provides academic and fiscal autonomy to all schools in the system; 3) Science and Mathematics standards are fully in place; 4) certification of new teachers is now based on skills and requires a test to demonstrate proficiency in both Science and Mathematics; and 5) a career ladder will allow teachers to be promoted on the basis of teaching performance and academic achievement of their students.

The PRDE has increasingly invested more funds to improve reformed schools and support their needs. For instance, Eisenhower Funds have been directed to professional development that follows the PR-SSI reform model for professional development. Also, the importance of technology to education in the 21st. Century was recognized by the Governor of Puerto Rico and by the PRDE. An investment of more than $0.8B was made to provide all teachers with laptop computers and every 17 students with access to at least one personal computer;  libraries are being digitalized; and access to the Internet is being provided to all schools.

The PR-SSI's Research Approach to Evaluation and Assessment

Conventional approaches to evaluation are not readily adaptable to the evaluation of systemic educational reforms because of their uniqueness and complexity. As has been evident throughout this paper, a reform needs to tackle simultaneously multiple elements within the system to be effective in transforming the system according to the reform’s intended goals. Program leaders and evaluators often have a very short time frame to identify and analyze what is really taking place within a reform and to show the impact of the reform’s interventions in transforming the system so that attribution can be clearly established.

Given the challenges of evaluating a systemic educational reform in Science and Mathematics and the dearth of available information to guide the process, the PR-SSI followed a research approach to document the reform’s progress in achieving its goals (Dávila, 1996).  In close collaboration with the reform’s leadership and participants at all levels, the evaluation and assessment of the PR-SSI focused on designing research questions (Dávila & Gómez, 1994; 1995; Dávila, Gómez & Vega, 1996 among others); these questions in turn led to the formulation of more questions as the reform evolved.  Therefore, the PR-SSI’s evaluation and assessment became an active element of the reform in its own right. 

The information provided by the evaluation of the PR-SSI is being   used to monitor progress, design quality control mechanisms, and retool the initiative as necessary (Puerto Rico Statewide Systemic Initiative, 1997; Puerto Rico Statewide Systemic Initiative, 1998). Building from the literature that suggests the use of multiple measures to triangulate results to identify trends, the PR-SSI has used combinations of qualitative and quantitative approaches to look at the changes taking place in the K-12 education system in Puerto Rico (Laguarda, Goldstein, Adelman, & Zucker, 1998). These approaches have included the use of classroom observations, semi-structured interviews, questionnaires, and standardized tests among other data collection strategies.

Measuring student academic achievement in Science and Mathematics has been a major focus of the PR-SSI and of its evaluation and assessment design. Looking at student academic achievement allows the PR-SSI to “take the temperature” of its overall reform strategy while focusing on the most important element of the system: the students.

The PR-SSI looks at student academic achievement from three different yet complementary perspectives: the classroom level,  the reform program, and the system. The teacher, who learns about alternative assessment strategies during professional development sessions designed for this purpose, looks at student performance in the classroom and uses this information to refocus the content and strategies of daily lessons to strengthen student learning. The program looks at student academic achievement trends across schools and across cohorts of participating schools using instruments such as the tests that include public-released items from NAEP and TIMSS. The system uses data provided by entities such as The College Entrance Examination Board (CEEB) to look at the impact of the Science and Mathematics reform on the overall system. The discussion that follows will be focused on evidence of success from the perspective of the program and of the system.

Measurement of Value-Added by the Reform On a Yearly Basis

A significant indicator of the success of a reform is whether its interventions have added value to the educational enterprise. One way to look at the value-added by the reform is to look at gains over the course of one school year by measuring student academic achievement at the beginning and at the end of that same school year (Dávila, 1999).

To address this need, the PR-SSI designed tests, in partnership with the CEEB, to measure gains in student mastery of Science and Mathematics content and development of thinking skills over the course of one year for fourth, eighth, and eleventh grades using public-released items from NAEP and TIMSS. These tests allow the PR-SSI to: (1) measure what the students are expected to know when they finish a grade; (2) provide feedback at the school level about student academic achievement; (3) address specific professional development as well as academic and technical assistance needs at the school level; and (4) compare student academic achievement against national and international benchmarks. 

The PR-SSI staff designed the pre/post tests in collaboration with the CEEB to ensure that the tests would measure what is valued by the reform. The tests are administered and scored following the CEEB’s protocols to ensure the security of the tests and the validity of the results. The CEEB has also equated the scales of the PR-SSI’s pre/post tests to those of the TIMSS to allow for comparisons against international benchmarks. 

The PR-SSI’s Science and Mathematics pre/post tests include multiple-choice questions and open-ended questions. Multiple choice questions are designed to test for mastery of content with a heavier emphasis on memory while open-ended questions measure thinking and reasoning skills within each discipline.

Since the PR-SSI’s unit of change is the school, pre/post test scores are reported at the school level. School principals and teachers receive the relative ranking of their schools’ performance within their immediate geographical region without any identifying information of the other schools. The purpose of sharing this information is to help school principals and teachers to take a critical look at student learning in their schools particularly in terms of item categories such as those defined by NCTM and NRC standards. Teachers and school principals receive direct assistance from the PR-SSI to strengthen the teaching/learning experience based on the needs of the students as evidenced by the results of these program assessment instruments.  The 1998-99 administration of the PR-SSI’s pre/post tests showed statistically significant gains in Science and Mathematics as measured by the multiple-choice and open-ended questions.  Since the overall score distribution graphs behaved in a similar fashion across grade levels and subject matters, our discussion will now be focused on specific examples.

Figure 7 presents the distributions of pre/post test multiple-choice student scores in Mathematics for the fourth grade. On the average, all students made significant gains thus validating the tenet that the PR-SSI is for all students. Figure 7 essentially shows a normal distribution for the pre-test, however, the post-test distribution shows a significantly skewed distribution towards the higher end of the scale whose average is established at 500 in alignment with the TIMSS scale. The highest pre-test scores were less than 750 while the highest post-test scores were in the 850’s which shows a gain of about 100 points for some students; these results show that the teaching/learning environment in PR-SSI schools permits all students to express their full potential.

 

Figures 8 and 9 present the average score distributions by school for the Mathematics multiple-choice and open-ended questions for the fourth grade. Again, the average score distributions shown in Figures 8 and 9 for the pre-test suggest a normal distribution while the average score distributions for the post-test suggest distributions skewed towards the higher scores. However, Figures 8 and 9 show post-test gains of more than one standard deviation relative to the pre-test mean scores. This result makes a strong case for the implementation of the PR-SSI’s “Whole

 

Figure 9

 
School Approach” to transform the teaching/learning culture of the schools which promote positive learning environments for their students.

 

When we compare the gains shown by the scores in the multiple-choice questions with those in the open-ended questions in Figures 8 and 9, we can see higher gains at the school level in the open-ended questions than in the multiple-choice questions. This result can be attributed to the PR-SSI’s emphasis on the development of thinking and communication skills in alignment with NCTM standards.

A Longitudinal Analysis to Determine the Long-Term Impact of the Reform on Student Achievement

Admission to college is an important indicator of the success of the systemic educational reforms being implemented. The PR-SSI uses student scores in the CEEB’s college admissions test in Mathematics Reasoning to look at the long-term impact of the reform in student academic achievement; this test has shown a 0.87 correlation with the SAT according to studies conducted by the CEEB. SAT data from 1998 shows gaps in performance between  majority  students and Hispanic students that are similar to those of students from private schools and public schools who took the CEEB; these gaps are of 62 points in the SAT and of 71 points in the CEEB (see Figure 10). 

 

 

The PR-SSI formulated a series of research questions to: 1) determine the long-term impact on the reform on student academic achievement; and 2) examine whether the PR-SSI was contributing to reducing the typical performance gap between students who attend public schools and students who attend private schools. The main questions were: 1) How is the performance of PR-SSI students in the Mathematics Reasoning test of the CEEB after three years and six years of participation in the reform; 2) How does the performance of PR-SSI students in the Mathematics Reasoning of the CEEB test after three and six years of participation in the reform compare with the performance of students from non-participating public schools and against the performance of students from private schools? (Figure 11).  



           

The PR-SSI analyzed CEEB data from four groups of students: (1) students from regular public schools (i.e., non- participants in the reform); (2) students who attended private schools; (3) students who attended PR-SSI

Figure 11

 
middle schools but who graduated from unreformed high schools thus they had three years of experience with the reform; and (4) students who attended PR-SSI middle schools and high schools thus they had six years of experience with the reform. The results presented in Figure 10 show that students who had three years of experience in the PR-SSI reduced by 32 points the performance gap between students from private schools and students from non-participating public schools. These results also show that the students who had six years of experience in the PR-SSI outperformed by 58 points students from private schools who tend to be the highest performing students in standardized tests.  Figure 12 shows that these trends  have been consistent over two years even when the number of the cohort of PR-SSI students who graduated from high schools and took the CEEB tests more than doubled between 1998 and 1999. 

The data shown provide clear and consistent evidence of the long-term impact of the transformed teaching and learning culture on student academic achievement. These data make a compelling case for the use of a systemic reform approach to improve the schools where the majority of the underserved students are enrolled in order to improve their learning experience and mainstream them into the educational pipeline.

The New York City Experience: Testing the PR-SSI’s Education Systemic Reform Model in Other Settings

The PR-SSI model is based on sound educational principles and the theory of social change in complex social systems, therefore, it is not culturally specific. This model was tested in Puerto Rico’s public schools, where over 80% of the students’ family income is below the federally defined poverty level and the schools show many other characteristics of inner city schools in the mainland. The PR-SSI’s systemic reform has shown that it can significantly improve the academic performance of underserved students vis-à-vis the performance of the more affluent students from Puerto Rico’s private schools that are more akin to suburban schools in the mainland in the socio-economic profile of their students and in their access to resources. Thus, the PR-SSI was uniquely poised to test the hypothesis that its systemic reform model for schools with underserved populations can be transferred and adapted to inner city schools in the mainland and succeed in improving student performance in Science and Mathematics.

The PR-SSI competed and secured funds from the Federal Department of Education (USDE) to test its systemic reform model in twelve New York City Public Schools in the South Bronx through a project entitled “Puerto Rico/New York City Educational Linkages Demonstration Project”. The project has already initiated professional development for participating teachers and begun the implementation of the “Whole School Approach” of the PR-SSI reform in the first three of these schools. The first three schools are being groomed as possible sites of future Dissemination Centers to scale-up the reform in other New York City schools; the remaining nine schools began their professional development experience this spring. Although it is too early to show quantitative improvement in student academic achievement since, in the PR-SSI’s experience, it takes about three years before full implementation of the “Whole School Approach” is completed and improved student academic achievement can be observed. Qualitative data collected during classroom visits and semi-structured interviews with students, teachers, and school principals suggests that the systemic reform effort is developing in the same manner that has been observed in Puerto Rico’s schools.

Congressman José Serrano, who represents the South Bronx districts where the first three schools are located, was sufficiently impressed with the progress of these schools that he secured the Congressional approval of an additional $750,000 in funding for the project. Furthermore, representatives of the United Federation of Teachers (UFT), are supporting the implementation of the project and are discussing with PR-SSI staff further possibilities of collaboration in New York City particularly in the form of professional development activities and support of UFT Teacher Centers.

The PR/NYC Educational Linkages Demonstration Project is an experiment in progress. If it succeeds in demonstrating that the educational systemic reform model developed and pioneered by the PR-SSI is fully adaptable to inner city schools with large underserved populations, then a mechanism will be available to ensure that inner city schools can provide access to the full educational pipeline to underserved students.

Completing the Reform Cycle: Links of K-12 Reform with Undergraduate and Teacher Preparation Reform

Completing the Reform Cycle

Thus far, a description and elucidation of the principles that govern  systemic educational reform has been made for a relatively self-contained K-12 system and evidence has been provided to demonstrate that the implementation of the systemic educational reform model can improve the academic performance of underserved students in Science and Mathematics. Yet systemic thinking requires that the links and interdependence between K-12 and the rest of the educational system are considered. Only then  the overall effectiveness and efficiency of the K-16+educational pipeline is to be optimized. The University of Puerto Rico, through its Resource Center for Science and Engineering, has done just that. Furthermore, the UPR has actively undertaken systemic reform of the 13-16+ of the  pipeline through the NSF-sponsored Puerto Rico Louis Stokes Alliance for Minority Participation (PR-LSAMP) and is improving teacher preparation programs through the NSF-sponsored Collaboratives for Excellence in Teacher Preparation (PR-CETP); these programs complete the reform cycle. The PR-LSAMP has undertaken the mission of improving the effectiveness and efficiency of undergraduate education in Science, Mathematics, Engineering, and Technology (SMET) in order to increase the number of undergraduates who finish their B.Sc. and go on to obtain a Ph.D. in these fields. The PR-CETP has brought together the faculties from the Schools of Education and Natural Sciences to revise their curricula in order to prepare the future teachers to perform effectively in the PR-SSI reformed schools. The RCSE serves as the virtual organization that orchestrates this complex multisystemic reform effort.

SMET Undergraduate Reform

The approach followed by the PR-LSAMP to achieve multi-campus and multi-institutional systemic reform in the 13-16+ section of the pipeline has followed the same principles and similar strategies to those utilized in the K-12 reform. The reform cycle illustrated in Figure 4, fully applies to the PR-LSAMP undergraduate reform approach. But, contrary to the K-12  approach, no effective unit of change could be identified in the higher education institutional structure. Academic departments, the obvious choice for unit of change, are too diverse and idiosyncratic in nature to serve as effective reform agents.

Instead, a two-prong approach to systemic reform was followed. A cadre of reform inclined faculty willing to experiment with change was identified and nurtured to pioneer the educational reform of university SMET courses, curricula, and teaching methodologies and Chief Executive Officers (CEO’s ) – presidents, chancellors, vice-presidents, and some deans – were persuaded with the evidence of successful reform efforts to support the institutional cultural transformation needed to make the undergraduate educational pipeline more effective and efficient. In doing this, the A3 approach was followed.

Reformed courses and curricula and innovative teaching methods pioneered by the cadre of reform-oriented faculty followed the same educational and pedagogical principles used in the K-12 reform (Rothman & Narum, 1999; American Association for Higher Education, 2000). Emphasis was put on: student-centered learning; inquiry and discovery approach; undergraduate research; less content and more development of understanding; use of technology; and the use of innovative classroom assessment methods. Pilot tested courses, curricula, and teaching methodologies were assessed, and, using the A3 approach, CEO’s were persuaded to invest institutional resources in scaling-up the programs and courses into full implementation with varying degrees of success. There are three major reasons for this limited success: 1) college level faculty seems to be more resistant to change than K-12 faculty; 2) academic accountability procedures at universities are more indirect and many times totally ineffective; 3) the considerable degree of autonomy of departments in academic matters is a major impediment to systemic reform.

Since the overriding goal of PR-LSAMP is to increase the number of students who successfully complete their undergraduate degrees and go on to obtain Ph.D.’s in SMET areas, a set of systemwide metrics to measure the achievement of this goal were developed. These metrics included longitudinal studies of: graduation rates; time to completion of degree; change in the number of degrees conferred, and what was called the “Index of Course Efficiency”(ICE). This last metric seeks to measure the probability of a student to successfully complete key SMET courses that are usual bottlenecks in their progress through the undergraduate pipeline, the so-called “gatekeeper courses”. This set of measures that assess academic progress have direct impact on the cost of higher education and, for this reason, have a major impact on the academic CEO’s and their setting of institutional priorities. Those metrics were incorporated into the Institutional Research process of some academic institutions to help to guide institutional planning and resource allocation.  CEO’s responded much better to these indicators and, therefore, the indicators are more effective in driving the A3 process for institutional reform.

The overall results of the PR-LSAMP education reform have been impressive. The number of SMET B.Sc. degrees conferred by the University of Puerto Rico has increased from 1,200 to 2,100 in a seven year period while, during the same period (displaced by five years to compensate for the average time to complete the degree) absolute enrollment of SMET students at UPR decreased (see Figure 13). This increase in degrees conferred while overall enrollment is decreasing is a clear indication of increased academic efficiency. Moreover, graduation rates for Science and Mathematics have gone from 46% to 60% in the same time period and, for engineering, they have risen from 58% to 75%. The number of students who go on to complete their Ph.D.‘s in SMET disciplines in the UPR’s Río Piedras Campus is one in ten, a number that puts the UPR among the top performers in the nation. Thus, although the objective of fully transforming the teaching/learning institutional culture of the university has proven to be a more elusive systemic reform objective than has been the case for K-12 reform, overall improvement of effectiveness and efficiency of undergraduate education has been achieved.

Figure 13

Teacher Preparation Reform

The PR-CETP has the mission of transforming teacher preparation programs at six different institutions of higher education and attuning them to the K-12 reform. The PR-CETP is a spinoff of the PR-SSI which initially included as part of its mission reforming teacher preparation programs. The design of the PR-CETP has also followed the reform cycle (see Figure 4) and has used the principles of systemic thinking. An analysis of existing teacher preparation programs by the PR-CETP revealed that there was almost no communication or effective academic interactions between the Schools of Natural Sciences and Education; this lack of interaction has resulted  in a poorly designed curriculum and sequence of courses for the teacher preparation program. For this reason, the PR-CETP designed its reform around a joint Natural Sciences and Education revision of the teacher preparation courses and curricula.

The Science and Mathematics core courses have been revised and the successfully pilot tested gatekeeper courses developed by PR-LSAMP, used to improve the SMET preparation of future teachers. Science and Mathematics methodology courses are being jointly revised by Natural Sciences and Education faculty. The education and pedagogy courses are being revised using the same educational principles used in the K-12 reform and teacher preparation students will do their practicum in K-12 reformed schools under the guidance of PR-SSI trained teachers.

The PR-CETP staff is using the A3 strategy to assess the pilot tested reformed courses and redesigned curricula and to persuade higher education leaders to support the reform effort. If successful, the PR-CETP will provide the highly trained humanpower needed to sustain the K-12 reform.

Final Comments

Over the years, the UPR’s RCSE has pioneered a series of reform strategies that have been successful in transforming teaching and learning in Puerto Rico throughout the K-16+ continuum. In this paper, we have shown evidence of the success of these strategies in achieving the goals of the reform which is primarily directed towards mainstreaming traditionally underserved students.  We would now like to highlight some of the lessons that we have learned in this process as well as some of the issues that still remain unanswered.

One of the main lessons that we have learned is that reform strategies cannot be developed and implemented in a vacuum. Without a solid theoretical and philosophical foundation to guide the development of these models and strategies, a reform is likely to be a project instead of a true change in institutional culture.

Another lesson that we learned is to “start small, and, then, scale-up”. This lesson enabled us to carefully monitor our efforts, using the A3 cycle,  so that we could make the necessary corrections in our design to ensure quality control. As we learned about what worked and about what did not work, we also developed the necessary human capacity to carry the reform forward and to sustain it beyond its funding period. We also learned about what were the key elements of the reform that could be transferred elsewhere as we have shown with our examples from the Puerto Rico/New York City Educational Linkages Demonstration Project.

Another important lesson entails the importance of establishing and maintaining strong connections between the different sectors of the educational system to be truly systemic. Without those connections, our reform would have been another fragmented effort to solve a systemic problem that would have been destined to fail by design. The virtual organization and pipeline models described in this paper are our answers to these issues which include building strategic alliances to support the efforts.  

Our systemic educational reform has transformed itself from its early days. Our experiences have led us to understand and appreciate that “what got me here today will not get me where I need to be tomorrow” because of the dynamic nature of social systems. Therefore, we are constantly looking for new and better ways to catalyze change in the system. We are also embarked on a constant search for answers to questions such as the following:

How can we learn more about the value-added by the reforms?

How can we improve pre/post test designs to look at the value-added by the reforms?

Why do we still focus so much of our attention on teaching instead of on learning?

What can we do as researchers and educators to learn more about the process of learning?

Why is student performance higher in open-ended questions than in multiple-choice questions?

We invite you to think about these questions and to formulate additional questions to guide our collective research agenda.

 

References

AAHE (2000). Targeting curricular change: Reform in undergraduate education in Science, Mathematics, Engineering, and Technology. Washington, D.C.: Author.

Ackoff, R.L. (1999). Ackoff’s best: His classic writings on management. New York, New York: John Wiley & Sons, Inc..

Corcoran, T.B., Shields, P.M., & Zucker, A.A. (1998). The SSI’s and professional development for teachers. Menlo Park, California: SRI International.

Culotta, E. & Gibbons, A. (Eds). (1992) Minorities in Science: The pipeline problem. Science, 258, 1175-1235.

Dávila, N. (1999). Assessing student outcomes. In N.L. Webb (Ed.) Evaluation of systemic reform in Mathematics and Science: Synthesis and proceedings of the Fourth Annual NISE Forum. Madison, Wisconsin: National Institute for Science Education.

Dávila, N. (1996). Design and use of an interactive evaluation model to assess systemic change in Science and Mathematics education. Paper presented at the ASCD 51st. Annual Conference, New Orleans, Louisiana.

Dávila, N. & Gómez, M. (1994). Assessment of the impact of a new curriculum on systemic change. Paper presented at the AERA Annual Meeting, New Orleans, Louisiana. ERIC DOCUMENT 390 307

Dávila, N. & Gómez, M. (1995). Evaluation of school-based regional dissemination centers as scale-up mechanisms for systemic educational reform in Science and Mathematics. Paper presented at the AERA Annual Meeting, San Francisco, California. ERIC DOCUMENT 310 702

Dávila, N. Gómez, M. & Vega, I.Y. (1996). Evaluating the transformation of the teaching/learning culture of schools involved in systemic Science and Mathematics educational reform. Paper presented at the AERA Annual Meeting, New York , New York. ERIC DOCUMENT 395 803

Kimmelman, P., Kroeze, D., Schmidt, W., van der Ploeg, A., McNeely, M., & Tan, A. (1999). A first look at what we can learn from high performing school districts: An analysis of TIMSS data from the first in the world consortium. Washington, D.C.: United States Department of Education.

Laguarda, K.G., Goldstein, D.S., Adelman, N. E. & Zucker, A.A. (1998). Evaluation of the National Science Foundation’s Statewide Systemic Initiative (SSI) Program. Assessing the SSI’s impact on student achievement: An imperfect science. Menlo Park, California: SRI International.

Loucks-Horsley, S., Hewson, P., Love, N., & Stiles, K.E. (1998). Designing professional development for teachers of Science and Mathematics. Thousand Oaks, California: Corwin Press.

Mervis, J. (1998). Mixed grades for NSF’s bold reform of statewide education. Science, 282, 1800-1805.

Oakes, J. (1990). Lost talent: The underparticipation of women, minorities, and disabled persons in Science. Santa Monica, California: The Rand Corporation.

Puerto Rico Statewide Systemic Initiative (1997). Annual Report. San Juan, Puerto Rico: Author.

Puerto Rico Statewide Systemic Initiative (1998). Annual Report. San Juan, Puerto Rico: Author.

Rothman, F.G. & Narum, J.L. (1999). Then, now, and in the next decade: A commentary on strengthening undergraduate Science, Mathematics, Engineering, and Technology education. Washington, D.C.: Project Kaleidoscope.

United States Department of Education (1998). The educational system in Japan: Case study findings. Washington, D.C.: Author.