Educational Effectiveness Research: Origins, Aims, and Evolution

Educational Effectiveness Research (EER) began in reaction to studies by Coleman et al. and Jencks et al. which found that schools were less influential to student outcomes than students’ backgrounds or natural abilities (Coleman 1966; Jencks 1972). From the inception of the discipline, EER had to defend against the claim that “schools [have] little effect on the outcomes of their students” (Reynolds 2014, p3). As a result, EER underwent distinct periods of change in which researchers sought first to prove that schools did in fact have effects on students, then to identify the effects themselves and investigate their underlying causes. As Creemers and Kyriakides expound in their dynamic theory of educational effectiveness, EER is at its core a study of change over time (Creemers 2008). According to them, EER examines the interplay of various contextual factors both inside and outside the classroom in order to evaluate whether schools are actually accomplishing what they exist to accomplish. Central to its goals as a discipline is the pursuit of answers to critical questions such as: What is education intended to accomplish? Who is qualified to decide this? How do we measure, and subsequently evaluate, the extent to which education is successful in fulfilling its goals? What factors should we consider in our evaluation? How do these factors interact with each other? And so on. Here, I will trace EER’s evolution as a discipline, examining in particular the ways it has expanded to account for a vast, complex array of variables behind school success—and, perhaps most importantly, why it has done so.

There are two general dimensions of school effectiveness: quality, determined by how a school scores in comparison to other schools, and equity, determined by how well a school compensates for input characteristics (Reynolds 2014). Each of these dimensions is primarily concerned with a school’s ability to help students make gains on cognitive skills as opposed to facilitating affective learning (development of social skills, value instilment, identity building, etc.). However, because a school’s aims normally extend beyond simply facilitating cognitive development, Reynolds argues that school evaluations should be based on criteria that assess multiple aspects of schooling—which this can be difficult, as affective outcomes are more difficult to measure precisely than their cognitive counterparts (Reynolds 2014). Whereas early EER focused on investigating factors such as physical resources or class sizes, which could easily be controlled by the one-size-fits-all decisions of legislators, recent research – such as John Gray’s 2014 review of the most critical influences on student learning – suggests that the factors of school success are in reality much more context-dependent (Ilie 2016). While these factors range from high expectations of student achievement to strong educational leadership to creating positive school cultures, the expression of each factor will look vastly different depending on how its implementation is carried out in different contexts (Edmonds 1979; Sammons 1995).

Whether or not a school is “effective,” according to Madaus et al., depends on whether it accomplishes what it sets out to do (Madaus 1980). However, what any given school sets out to do is inevitably different according to the context in which it exists—a context which includes its surrounding community, its district, and even its country. Reynolds et al. found that while teacher-level factors and pedagogies were similar across different countries, for example, school-level policies and practices vary immensely according to the societal customs and contexts of different locations (Reynolds 2014). Articulating the purpose of education is a fundamental concern of EER because pinpointing what is and what isn’t an “effective” school can only be accomplished by assessing how well the goals of schooling are actually met. Since the broader context in which a school and its students exist significantly influences not only these aims but also how well they are implemented, EER methodology has increasingly shifted to reflect the complex interactions of multiple contributing factors – as I will describe.

The move toward “evidence-based education,” combined with the fact that EER emerged as a response to an existing scientific claim, made the need for empirical evidence increasingly apparent (Bosker 2011). As a result, EER produced primarily quantitative data during its early stages, which demonstrated the relationships between educational factors without positing explanations for these relationships. Lack of explanation made it difficult for policymakers and practitioners to access EER findings, however, which one could argue has led the discipline to increase the number and scope of qualitative studies in recent years (Reynolds 2014).

Because EER has developed to more accurately represent the complex reality of education systems, accounting for nested levels of influence, its methodological focus has tended toward models capable of taking numerous factors into consideration. Multilevel modelling (MLM) in particular emerged as a way for researchers to measure the relationships between “multiple variations, differential effects, and cross-level interactions” in a way that acknowledges clustering within datasets, which multiple regression analyses largely failed to account for (Reynolds 2014, p7). Early MLM focused solely on direct educational effects—within the classroom, for example, researchers considered the effects of teachers on students but left the reverse relationship unexamined—and neglected the study of the interactions between levels of influence, such as the effects of school or district policy on classroom dynamics and vice versa (Reynolds 2014). Cross-classified models, in addition to MLM, enable researchers to examine the ways in which school effects correlate with student intake characteristics. These models make it possible to investigate the effectiveness of a school for students from different socioeconomic or ethnic backgrounds, of different genders, and with differing levels of prior attainment—or various combinations thereof.

Additionally, EER’s emphasis has shifted from cross-sectional to longitudinal studies as it seeks to measure the value added (VA) to a student’s education by any given school. In keeping with EER’s focus on the ways in which different circumstances influence student outcomes, VA models allow measurement of student progress over time rather than achievement at any one point; contextual value-added (CVA) models take into account not only student progress but also their unique input characteristics. Each of these models allows EER to measure school effects more realistically, accounting for more and more factors in the complex reality of the education system.

EER makes increasing use of meta-analysis to include different contexts, increase sample sizes, and reach better-supported conclusions more generally, but these conclusions are neither blueprints for action nor are they generalizable beyond their specific contexts. Ironically, perhaps one of the greatest frustrations with EER arises from the importance it places on context. While the results EER yields are based in comprehensive, wide-reaching studies, the practical implications of these results vary from situation to situation. Making the findings of EER accessible to policymakers and practitioners is often difficult for this reason; since legislators even within the same country are responsible for large groups of constituents—and, as a result, schools that exist in different contexts—it is easy to see the appeal of one-size-fits-all reforms, and to understand the resulting frustration when EER inevitably fails to prescribe them. What works in one school may look vastly different from what works in another, despite the fact that both share the aim of increasing “effectiveness”—though, as previously noted, even this can mean different things to different schools.

Despite its origins as a discipline defending against a single claim, EER has expanded to become a truly international inquiry. Its primary focus remains understanding the ways in which different factors affect not only each other but overall student outcomes; simultaneously, it embraces the importance of learning by comparison and examining schools in different contexts to learn what works in different situations and why. Though it may be frustrating to reformers that EER findings can be interpreted and implemented differently in different situations, the discipline’s increasing interest in individualized, contextual analysis is ultimately more effective in accurately reflecting the current state of our education system. After all, the system itself exists to serve individual students, and the beautiful, difficult reality of being human is that no two experiences are identical.

 

References

Bosker, Roel J. “From Educational Effectiveness to Evidence Based Education.” International Congress for School Effectiveness and Improvement. Limassol. Address.

Coleman, James, Ernest Campbell, Carol Hobson, James McPartland, Alexander Mood, Frederic Weinfeld, and Robert York. “Equality of Educational Opportunity.” Washington, DC: US Government Printing Office, 1966.

Creemers, Bert and Leonidas Kyriakides. The Dynamics of Educational Effectiveness: A Contribution to Policy, Practice and Theory in Contemporary Schools. London: Routledge, 2008.

Edmonds, Ronald. “Effective Schools for the Urban Poor.” Educational Leadership (1979): 15-24.

Ilie, Sonia. “School Effectiveness Research: Aims, Methods, and Contextual Translatability.” University of Cambridge, 2016. Presentation.

Jencks, Christopher, Marshall Smith, Henry Ackland, Mary Jo Bane, David Cohen, Herbert Gintis, Barbara Heyns, and Stephan Michelson. Inequality: A Reassessment of the Effect of the Family and Schooling in America. New York: Basic Books, 1972.

Madaus, George F., Peter W. Airasian, and Thomas Kellaghan. School Effectiveness: A Reassessment of the Evidence. New York: McGraw-Hill, 1980.

Reynolds, David, Pam Sammons, Bieke De Fraine, Jan Van Damme, Tony Townsend, Charles Teddlie, and Sam Stringfield. “Educational Effectiveness Research (EER): A State-of-the-art Review.” School Effectiveness and School Improvement 25.2 (2014): 197-230.

Sammons, Pam. Key Characteristics of Effective Schools: A Review of School Effectiveness Research. Rep. no. ISBN-0-85473-447-3. London: Office for Standards in Education, 1995.

 

written under the supervision of Dr. Sonia Ilie,
Education Access & Quantitative Methods