About the Study
[Below we describe the rationale behind The Study of Instructional Improvement and the major research questions the project sought to answer. The navigation links located on the left column introduce readers to the Research Team, SII Papers and Publications, and the Sample and Components descriptions, along with contact information, related links and a note of thanks to our study participants and sponsors.]
Reforms in the federal Title I program, passage by Congress of the Comprehensive School Reform Demonstration Act, and Part F of the No Child Left Behind (NCLB) act, have focused attention on what many analysts now call “whole-school” or “comprehensive school” reform. This emerging conception of school improvement stands in sharp contrast to previous initiatives, especially efforts that sought to improve instruction and student achievement in high-poverty schools through isolated activities such as the adoption of new curriculum materials, the provision of brief training to teachers, or the provision of compensatory instruction to low achieving students within schools. A great deal of evidence suggests that these isolated efforts did little to markedly improve instruction and student achievement in schools, especially high poverty schools. As a result, efforts at comprehensive school reform sought to address the problem of instructional improvement more broadly. Gone were attempts to focus change on isolated elements of schooling. Instead, efforts at comprehensive school reform sought to improve the instructional capacity of entire schools, and to do so in ways that involved systematically changing many different (and interconnected) elements of instruction and instructional capacity in schools and classrooms.
One interesting outcome of this movement was the emergence of a large number of comprehensive school reform (CSR) interventions. Around the year 2000, more than 200 such interventions were operating in the United States, and interventions were adopted in more than 10,000 schools around the country. The emergence and widespread adoption of these interventions offered the education community an unprecedented opportunity to examine new conceptions of instructional improvement and to investigate empirically how these new conceptions were being put into practice. Unique opportunities for research were available because these school improvement interventions were based on a variety of designs for instructional improvement and because these designs were being put into practice in a wide range of school communities. Thus, perhaps more than ever, the education community was finally in a position to take a serious and sustained look at whole-school approaches to instructional improvement: to examine schools pursuing different, systemic designs for improving instruction and student achievement, to examine how implementation of these designs was affected by different patterns of external assistance, and to see how processes of instructional change unfolded in a variety of school, community, and policy environments.
Although comprehensive designs for instructional improvement appeared promising, high quality research on the problem of instructional improvement remains scarce. At the outset of the SII project, little was known about the alternative designs for whole-school initiatives, instructional improvement or about the various strategies that external agencies could use to promote substantial and sustainable instructional change. Also, few longitudinal studies tracing the implementation of alternative designs for instructional improvement in local schools existed and little research existed examining how implementation of these designs varied across different state and local policy environments. More importantly, few studies looked inside classrooms to probe the effects of interventions on the dynamics of teaching and learning in particular subject areas, or to understand what teachers need to learn in order to make changes in their practice. Finally, there was a lack of solid empirical research on the effects that whole-school approaches to instructional improvement could have on student achievement, especially for students attending diverse schools, coming from different family backgrounds, and living in different kinds of communities.
The Study of Instructional Improvement
To meet the growing need for high-quality research on whole-school approaches to instructional improvement, researchers at the University of Michigan School of Education, in cooperation with the Consortium for Policy Research in Education (CPRE), conducted a large-scale, mixed method, longitudinal Study of Instructional Improvement to investigate the design, implementation, and effects on student achievement of three of the most widely-adopted whole-school school reform programs in the United States: the Accelerated Schools, America’s Choice, and Success for All. Each of these school reform programs sought to make “comprehensive” changes in the instructional capacity of schools, and each was being implemented in schools in diverse social environments. Each program, however, also pursued a different design for instructional improvement, and each developed particular strategies for assisting schools in the change process. In order to better understand the process of whole-school reform, SII developed a program of research to examine how these interventions operated and to investigate their impact on schools' instructional practice and student achievement in reading and mathematics.
The research program had three components:
- A longitudinal survey of 115 schools (roughly 30 schools in each of the three interventions under study, plus 26 matched control schools);
- Case studies of the three interventions under study; and
- Detailed case studies of nine schools implementing the interventions under study (plus 3 matched control schools).
Each of these research components is leading to separate reports and findings, although SII undertook these studies as an integrated program of research that examined issues related to whole-school, instructional improvement from multiple analytic and methodological perspectives. Across all components of the SII study, the research examined alternative designs for instructional improvement, alternative strategies for putting these designs into practice in local schools, and the extent to which alternative designs and support strategies promote substantial changes in instructional capacity and student achievement in reading and mathematics. A list of project publications can be found here.
All of this work had two main purposes. First, we wanted to know the circumstances under which different intervention designs and strategies could be expected to produce changes in particular elements of instructional capacity in schools; and second, we wanted to know which elements of instructional capacity, when present in schools, worked to produce higher levels of student achievement in reading or mathematics. Answers to these questions, we argue, provide powerful knowledge about how to successfully intervene in schools to promote instructional improvement.
In this online report, we focus on the longitudinal survey of schools. The primary media for reporting on the case studies of schools and case studies of intervention programs will be published manuscripts currently in preparation or press.
Longitudinal Survey of Schools
The most comprehensive component of SII was a large-scale, longitudinal, multi-survey study of schools. The use of survey research methods was intended to track the course of schools' engagement in comprehensive approaches to instructional improvement and to investigate the conditions under which this led to substantive changes in instructional practices and student achievement in reading and mathematics. The study design called for each school to participate in the study for a period of three years, although some schools voluntarily provided a fourth year of teacher, leader, and school-level information (no additional student-level data). Data were collected during the 2000-2001 through 2003-2004 academic years. During this time period, survey researchers administered questionnaires (available here) to teachers and school leaders on an annual basis in order to chart broad, organization wide changes in instructional capacity in these schools, including professionals' learning opportunities, the nature and focus of collegial interactions, and patterns of instructional practice. SII researchers also used a variety of other, more targeted data collection strategies to carefully chart the instructional experiences and academic learning of two cohorts of students (a cohort passing through grades K to 2, and a cohort passing through grades 3 to 5) in these schools. One important and innovative strategy for gathering information about instruction involved the use of language arts and mathematics instructional logs (available here) that teachers of cohort students completed on a daily basis (for selected students) in order to map the academic experiences of students as they pass through schools. Another strategy involved the use of twice-annual assessments to record students’ growth in academic achievement in both reading and mathematics.
In addition, survey researchers conducted interviews, primarily a telephone protocol (available here), with a parent or guardian of each cohort student in order to gather information on students’ family background and on students’ home and community environments. Researchers also gathered data from school leaders and others about the policy environments in which the schools are located.
These survey data can be used to address research questions in at least two analytic domains:
- One domain concerns patterns of change in schools participating in “whole-school” instructional improvement initiatives. Here, survey researchers can study: (1) the extent to which schools participating in different interventions develop different patterns of instructional capacity; (2) the consistency with which such patterns emerge among schools pursuing the same intervention; and (3) the extent to which patterns of change in instructional capacity are explained by features of intervention designs and support strategies, state and local policy environments, or initial conditions in schools adopting particular reform models.
- A second research domain concerns the extent to which schools’ participation in “whole-school” improvement produces changes that make a difference to student achievement in reading and mathematics. Here, survey researchers can carefully chart what students are taught in these two core school subjects and what they learn in these subjects, when such teaching and learning occurs, and how patterns of academic achievement in these subjects are affected by particular elements of instructional capacity in schools.
Case Studies of Schools
Another component of the research program involved the development of detailed case studies of a small number schools participating in the study. The case studies gathered observational, interview, and documentary evidence to better understand how instructional change processes unfolded in different school settings. Case studies were conducted in 12 schools operating in differently configured state and district policy environments. In each environment, researchers selected schools participating in one of the interventions under study as well as a “matched” control school.
These case studies explicitly recognized the diverse and multi-layered nature of policy environments in American education and characterized these environments in terms of the extent to which they provided coherent instructional guidance, incentives (of different kinds) to encourage local school improvement, and resources (of different kinds) to support school improvement. At each school site, SII researchers were especially interested in studying how teachers and administrators learned about the interventions they adopted, and how they interpreted and used the guidance and resources provided by interventions and other education agencies in their policy environments as they enacted these interventions. The hypothesis was that the processes of learning and interpretation in multi-layered and diverse environments have a significant influence on the enactment of instructional improvement activities in local sites.
Data from the case studies have been used to address the following questions:
- What are the various instructional improvement policies developed by state and local education agencies, how are these interpreted by school personnel, and what bearing does this have on how local personnel learn about and enact instructional improvement efforts?
- How do local school personnel perceive intervention designs and how do they learn to put these designs into practice? What is the relative influence of intervention-based learning opportunities compared to learning opportunities provided by other agencies, and how do local school personnel combine these learning opportunities to change their instructional practices? Do differently configured policy environments affect these processes of professional learning?
- Overall, how do local school professionals make sense of their environments, and how do they use (or not use) the various forms of instructional guidance, incentives, and resources in these environments to learn about and put into practice externally-designed, whole-school improvement interventions?
Knowledge gained through the case studies has been important to our research agenda for several reasons. First, case study results have been used to interpret and deepen our understanding of survey results, helping us interpret residual variance and/or deviant cases in our statistical analyses, thereby clarifying the complex interactions that explain instructional improvement operations and outcomes. Second, the case studies provided research-based information in forms that are highly conducive to professional learning, where concise and powerful “stories” provide powerful schemata for understanding the highly abstract and generalized findings of quantitative research. Most importantly, however, the case studies have been designed to contribute to the building of theories of intervention by placing attention squarely on the ways in which school personnel actively make sense of and enact instructional improvement efforts in different policy and organizational environments.
Case Studies of Interventions
A final task in our research program was to develop detailed knowledge about how the interventions under study were designed and how they functioned as organizations to manage their main task of intervening in local schools. These case studies are important in their own right, helping us understand the nature of the interventions we studied. But the case studies also were designed to help us understand some general problems associated with instructional improvement, especially designs for instructional improvement and strategies for change that seek to produce instructional improvement on a broad scale in American education, where local schools operate in a variety of social and policy environments. To build this knowledge, we examined the documents produced by the interventions under study, interviewed key personnel in central and regional offices of the interventions, and observed training and other professional development activities conducted by the interventions. The goals of this work were:
- To describe the curricular goals, instructional practices, and organizational structures and processes that each intervention seeks to foster in schools;
- To describe the approaches used by each intervention to recruit schools and support implementation in local settings; and
- To describe the management strategies developed by each intervention to evaluate and improve its services to local schools, to cope with the demands of “scaling up,” and to gain support for and/or neutralize disruptions to their work.
We used data from the case studies to chart key similarities and differences in the design and operations of the interventions under study, to analyze how different design features affect operating strategies, and to better understand the general problem of how intervention programs can work to devise and “bring to scale” a feasible scheme for improving instruction in local schools.
Project Endorsements
The Study of Instructional Improvement is endorsed by:
- The American Association of School Administrators
- The American Federation of Teachers
- The Association for Supervision and Curriculum Development
- The Council of Chief State School Officers
- The Institute for Educational Leadership
- The National Alliance for Business
- The National Association of Elementary School Principals
- The National Association of State Boards of Education
- The National Education Association
- The National School Boards Association
- New American Schools