The issue of sustainability

The study by Lynch et al. (2012) examines how three middle-school science curriculum units were implemented and scaled up in a large, diverse school system, and then assesses their sustainability four years after funding ended. This is done by retrospectively examining a research programme which was in function for six years from 2001 to 2007, and then revisi-ting its outcomes in 2011 in order to access the subject of sustainability. The programme in question, SCALE-uP (Scaling-up Curriculum for Achievement Learning and Equity Project), was designed to study the implementation and scale-up of three highly rated science curricu-lum units, each focusing on a particular area in science shown by research to be challenging for students. The programme followed a structure in which the individual units were first examined in order to determine if they were effective and equitable. If so, the units were scaled up in stages, then assessed in order to determine their effect under new conditions.

SCALE-uP was implemented in a very large metropolitan school system in the Central Atlantic region of the United States. The area was characterised by substantial cultural and socioeco-nomic diversity, as well as a long-standing tradition of top performances on state measures combined with high levels of participation in educational research activity. A quasi-experi-mental design with matched comparison groups and pre- and post-testing (at the beginning and end of each curriculum unit) was put in place, reaching a total of about 250,000 students and more than 120 science teachers. Pre- and post-test data consisted of student achievement results on science content assessments for both treatment and control groups under both small and large-scale conditions. These data were disaggregated by demographics in order to measure whether intervention effects were equitable. A range of qualitative data was also produced, including extensive classroom observations designed to measure fidelity as well as researcher notes and documentation of interactions with various different stakeholders.

During SCALE-uP, teachers in treatment schools received professional development during the summers prior to the introduction of the unit materials in classrooms. Refresher ses-sions were given during the school year and in the intervening summers between trials.

Comparison teachers received the district’s standard professional development to review the science content that was the focus of each portion of the study. Furthermore, treatment teachers were provided with teacher guides and classroom sets of student guides, as well as sets of the required laboratory materials (financed by research funds).

Overall, student outcomes from the implementation and scale-up processes were mixed.

Two of the science curriculum units were found to be effective and equitable as measured by assessments of students’ science knowledge in the small-scale scenario, and therefore went to scale. One of these units was tested for effects on a large scale, and was shown to be ineffective when taken to scale. Outcomes on the effects of teacher experience with this unit were surprising, in that students of less experienced teachers had the highest scores.

The authors perceive this as a likely consequence of high teacher mobility rates influencing the possibilities for ongoing professional development.

When examining the sustainability of the curriculum units four years after the end of research funding, the authors found that none of the curriculum units had been sustained within the school district. They explain this as a result originating partly from challenges pertaining to the individual interventions, and partly from environmental and contextual factors, in particular changes in the district policy climate. The description given by the authors is one of a policy landscape in constant flux in which educational goals, objectives, and obligations are ever-changing, leaving little space for interventions to settle – especially for those demanding rigorous implementation routines and time, such as the three science units contained in the SCALE-uP programme.

In conclusion, the authors point to intervention-specific as well as environmental/con-textual factors shown to respectively promote or hinder the implementation, scale-up, and sustainability of the curriculum units. The most important of these were implemen-tation fidelity, implemenimplemen-tation support, personal mobility (teacher turnover), conflicting interests, difficulties communicating research results, and an unstable policy climate.

Providing science teachers with implementation support in the form of materials and laboratory products seemed to act as a promoting factor.

• Implementation fidelity: The authors provide an example of how implementation fidelity has a vital significance for the effects (or lack thereof) of a given intervention. When

im-plementing one of the units, researchers were frustrated by poor and confusing results on student outcomes. They decided to replicate the trial in a different set of schools, pla-cing a strong emphasis on the fidelity of implementation. With this extra care taken to optimise fidelity, the intervention showed positive results and was allowed to go to scale.

• Staff mobility: Staff mobility was perceived to have a potentially undermining effect on professional development efforts.

• Conflicting interests: In some instances, there were conflicting interests between resear-chers and stakeholders, for instance when principals wished to use a research instrument designed for classroom observations of fidelity for teacher evaluations, which might potentially confound the research results and influence the perception of SCALE-uP.

• Difficulties communicating research: The authors faced some difficulties communicating research results to implementers, both for logistical reasons to do with large district size and because of difficulties in making research results understandable and usable for school personnel.

• Unstable policy climate: The unstable policy climate in the district, whereby new goals and standards were constantly having to be considered and where SCALE-uP curricu-lum units were competing with other interventions that carried higher stakes (and thus stronger incentives), undermined the possibilities for these units to succeed. Interventions demanding relatively large amounts of time and effort seemed not to be aligned with a policy climate in which more flexible and less demanding interventions, specifically those already embedded into the daily workings of schools, stood a better chance of survival.

The results reported by Leadbeater et al. (2015) come from two studies, both taken into consideration together and treated as a whole because they form part of the same research process, including results on start-up and take-up processes as well as considerations on the sustainability of interventions. The focus of the two studies is on describing experiences of discovering, actively evaluating and sharing the WITS programmes (Walk away, Ignore, Talk it out and Seek help) in rural Canadian elementary school settings.; Thus the investigation is of the processes that promote or inhibit early users’ discovery, understanding, incorpo-ration, and sharing of mental health promotion programmes. In addition to examining the start-up processes related to the pre-implementation phase of the WITS programmes in rural school districts, descriptions of opportunities and challenges for sustainability are also provided, illuminating the factors that influenced planning for continued use of the evidence-based WITS programmes two years after their adoption in eight rural Canadian elementary schools. (For more information see theme five, section 3.6.)

In this theme, the second part of the research process is reported, centring on the subject

of sustainability. Twenty-four individuals were interviewed in April 2013, all having pre-viously participated in at least one interview. These participants came from eight rural elementary schools. In this part of the study, fidelity was assessed by evaluating the use of eight core components of the programme, and the interviews focused on schools’ planning for sustainability.

Four central aspects were found to be of relevance: within-school influence, influences of the external context, programme characteristics and support, and the effects of variations in implementation.

• Within-school influences: Leadership teams were found to play active roles in sustai-ning programme use, for instance by promoting the programme or ensuring that new members of staff were trained. Accountability to administrators also hel-ped motivate teachers or other staff to continue using the programme. In addi-tion, uptake from teachers was seen as particularly salient to long-term stability.

Processes of embedding the programme and creating a common language were vi-tal to sustainability efforts, with WITS being made a part of the school culture, co-des of conduct, and everyday practice, as well as an integrated part of regular curriculum and teaching. It was found that all participating schools were main-taining the programme language even when they rarely used other programme components, leading to a description of the WITS language as “self-sustaining.”

Furthermore, ongoing communication and renewal of commitments were found to be required in order to sustain the programme, as well as a continued belief in the relevance and effect of the programme. Collaboration with a new community leader may also act as an antidote to programme decay, while being unable to sustain collaboration with a community leader could be a challenge to programme continuity. In this respect, turnover of children and staff was found to be a challenge as well as a promoting factor for sustaina-bility. In general, participating schools experienced high rates of staff turnover, including rotations of principals. A process of transferring responsibility for implementation was therefore necessary, with sustainability depending on the uptake of ongoing teachers, children, and parents. It was found that when teachers take ownership and involve multiple stakeholders, this helped to sustain WITS when a school principal left, for instance. How-ever, engaging or re-engaging multiple stakeholders over time also presented a challenge and required frequent renewal of commitments. The spread of the programme beyond single schools to numberous schools in a district also helped to overcome the challenges

connected with children changing schools within the district, because the children were familiar with WITS. This was seen as a particular strength of the WITS programmes.

• Influences of the external context: The results indicated that policy changes coming from outside the educational system could limit needed collaborations with the community.

The years in which WITS was being implemented were marked by two significant changes in educational policies initiated at the provincial level: one initiative appeared to enable WITS, while the second both reduced the time allowed for emphasising prevention and created uncertainty about the fate of WITS. Direct provincial support or endorsement by the British Columbia Ministry of Education could serve to enable the long-term sus-tainability of WITS. Overall, the continued growth of new programmes was seen as a potential threat to the long-term use of WITS: the district had a long previous history of waxing and waning programmes. However, it was the belief of some participants that WITS was resistant to infringing programmes.

• Programme characteristics and support: Considering the characteristics pertaining to WITS itself, the continued fit of programme resources was found to foster sustainability for some schools. Also, the access to ongoing support from WITS programme staff played a sustaining role. Meetings with the research team also helped to give a feeling of accoun-tability, leading to reflections on what could be done to improve WITS implementation.

• Effects of variations in implementation: Schools varied in their use of core WITS components.

Five schools implemented all eight core components, whereas three schools implemented only five or six components (the programme’s less complex aspects, such as the WITS language). These three schools had difficulty maintaining the components that pre-scribed specific teacher and community leader activities. At some schools, the fidelity of programme implementation varied over time and was cited as a concern related to sustainability planning. Some programme components, such as the WITS language, were easily maintained, even in the context of changing environments for programme delivery, whereas other core aspects were dropped, compromising both fidelity and the likelihood of sustainability. Even in schools with high fidelity, concerns were expressed that the programme might fade over time unless sustained efforts were made to energise the programme.

Overall, the institutionalisation of programme language and activities into school codes of conduct and everyday practice helped ensure sustainability. On the other hand, staff turnover, declining stakeholder investments, difficulties in engaging cross-sectoral support, unfavo-urable policy environments, and the perceived advantages of competing programmes were identified as potential threats to programme continuity. Mediating between these processes

of sustaining implementation or averting decay, the involvement of the school principal and the programme staff was required in order to promote the programme, provide opportuni-ties for training new staff, update and renew resources, and resist encroachment from new initiatives. In this sense, both the school and the programme staff needed to anticipate and respond to ongoing changes both within and outside the school. Sustainability appeared to depend on how well programmes, schools, and policy environments worked together over time to cope with staff turnover, policy and priority changes, and competition from other innovations. Sustainability planning relied on regular staff meetings, active leadership, support from provincial policies, and innovation within the programme. Implementation quality appeared to intersect with sustainability, so that schools with better adherence to programme components were more likely to describe a firm plan for continued use of the programme. The multisectorial approach of WITS helped to foster sustainability by embed-ding stakeholder roles in the programmes’ design, uptake, and implementation strategies.

To summarise. The above results were perceived as underscoring the lifecycle theory of sustainability, suggesting that sustainability can be understood as a function of ongoing planning for renewing implementation in school settings, with both self-sustaining cycles and cycles that require work and effort. The two studies overall point to the need for support not only for the initial uptake, but also across the programme’s life cycle. Sus-tainability is not merely a next step following high-quality implementation; it involves ongoing communication, evaluation, and re-commitment processes that must be antici-pated both by school principals and by programme developers.