5 years ago, only higher education was subjected to systematic external evaluation. EVA’s predecessor, the Danish Centre for Evaluation of Higher Education, was responsible for the systematic evaluation of all higher education. As mentioned earlier, the Centre was set up in 1992 to evaluate all higher education programmes within a 7-year period. When the Centre was established, the idea was that the Centre would start a new round of evaluations after the first 7-year period. This rotation principle would assure the follow-up to the first evaluations.
All evaluations were carried out according to the four-stage model, which includes: self-evaluation, an external expert panel (a steering committee), site visits and a public report. In addition, user surveys were conducted, and the reports of the external examiners were ana-lysed.
The guidelines for self-evaluation contained a number of themes that were relevant for the evaluation of all higher education programmes. The guidelines were, however, adapted to the specific context of a programme when necessary. The definition of quality was fitness for pur-pose. The evaluations thus took their starting point in the relevant legislation for a programme and the objectives formulated at programme level. Conclusions and recommendations were primarily made in this context.
The purpose of the evaluations was twofold:
Firstly, the evaluations aimed at providing the evaluated programmes with a contribution to their internal quality development. Development was primarily enhanced through a self-evaluation process where the programmes prepared a report on their strengths and weak-nesses in relation to the themes presented in the guidelines for self-evaluation. As all pro-grammes within one educational field were evaluated at the same time, the evaluations also sought to contribute to the development of the quality of the programme at national level where relevant. The reports nearly always contained recommendations to the Ministry of Edu-cation relating to the need for quality development at the national level.
Secondly, the evaluations also had a control function in as much as they provided the stake-holders with an insight into the quality of the evaluated programmes, through the publication of the evaluation reports. The control function was emphasised through the site visits and the
Educational Evaluation around the World 34
implementation of user surveys, e.g. among representatives of employers, students or gradu-ates.
The evaluations were thus characterised by having the same focus and the methods were, to a large extent, standardised.
The double purpose of the evaluations has been retained, and the evaluations still build on the same methodological elements, even though EVA’s mandate is broader than that of its prede-cessor.
In the near future the purpose of evaluations will continue to be twofold. They will still have to contribute to the quality improvement of the evaluated units in particular, and the evaluated field in general. Furthermore, the evaluations will continue to have a control function, as they inform stakeholders in the broad sense, both in Denmark and abroad, of the quality-status in the evaluated field.
The majority of evaluations will still use the objectives formulated at the national, local and institutional level as their starting point. However, due to international developments there will be increased interest in the results of education and in creating a higher degree of transparency of education quality across borders. There will be a need for quality definitions that are under-standable and acceptable across borders, and it will be necessary to develop other ways of describing quality than in terms of fitness for purpose. One of the means to obtain this higher level of transparency is through pre-defined criteria as the basis for evaluations; another is to focus on output measures, where it is easily identifiable whether expected targets have been met.
Last but not least, there will be an increased focus on competences as another means of mak-ing quality judgements comparable, i.e. what are the pupils or students capable of when they have gone through a particular programme at a certain level of the education system. This changes the focus of evaluation from the structures of education to the curriculum and the teaching methods, and the outcomes of teaching and learning.
There will be continued focus on the procedures set up by the institutions themselves to con-tinuously check and improve the quality of their activities and structures. Consequently, there will be a need for external quality assurance to check the effectiveness and sustainability of these internal mechanisms, and to undertake measures that give an input to the improvement activities initiated by the institutions through audit activities. However, that will not be suffi-cient, due to the European, or even international, demand for comparable assessment of qual-ity. Therefore, there will still be a need to initiate evaluation activities at subject or programme level, but with a transnational dimension.
With the increased international dimension in education, educational systems are becoming more and more complex. Therefore, there will also be a future need for broadness in the foci of evaluations, and in the corresponding methodological elements applied to assess these foci.
Educational Evaluation around the World 35
Northern Ireland
Stanley Goudie
Assistant Chief Inspector
The Education and Training Inspectorate, Northern Ireland
Values and purposes
The Education and Training Inspectorate (Inspectorate) provides inspection services and infor-mation concerning the quality of education and training in Northern Ireland (NI) to the follow-ing departments:
Department of Education (DE);
Department of Culture Arts and Leisure (DCAL);
Department for Employment and Learning (DEL).
The organisation is a unitary inspectorate, providing independent advice to all three depart-ments. The legal basis for the work of the Inspectorate is set out in the Education Reform (Northern Ireland) Order 1989 (Article Number 30).
The mission statement of the Inspectorate is, ‘Promoting Improvement (in the interest of all learners)’; and the organisation’s vision is, ‘The Inspectorate will be a highly regarded and influ-ential body, dedicated fully to the education and well being of all learners; its members will treat with respect and consideration, all those with whom they come in contact’.
The purpose of inspection is to promote the highest possible standards of learning, teaching and achievement throughout the education and training sectors, including teacher education within the five higher education institutions, but excluding the education provided by universi-ties per se. In serving this purpose, the Inspectorate:
provides objective professional evaluation, based on the findings of inspection, of the quality of learning and teaching, including the standards achieved by learners;
identifies and reports on educational developments and practices, and evaluates their influence on the quality of learning and teaching;
provides evaluative comment on the influence and outcomes of the policies of the three Departments on education and training services;
prepares reports on individual organisations, and overview reports on aspects of educa-tional and training provision.
The Inspectorate’s mission, vision, purposes, values and service standards are set out in its Charter for Inspection, published in October 2002.
The organisations to be inspected are identified by the Inspectorate. In recent years, however, those being inspected have become increasingly involved in determining the nature, scope, and timing of inspection. These important changes have been facilitated by the development of a range of types of inspection in addition to the more traditional general and focused inspec-tions. The range includes District Inspections, Two-Part Focused Inspections, Unannounced Inspections, Quality Assurance Inspections, and Self-Evaluative Follow-up Inspections.
Evaluation by the Inspectorate is understood to mean the making of informed judgements about the quality of provision and standards in education and training within and across North-ern Ireland. Such judgements are based mainly on first hand evidence of observed practice in a range of settings, including classrooms, lecture theatres, youth clubs and workshops, and are
Educational Evaluation around the World 36
supported, for example, by evidence gleaned from the scrutiny of examination data and docu-mentation provided by the organisations involved.
To help develop the quality of provision in education and training in NI, the Inspectorate has sought increasingly in recent years to foster an ethos of self-evaluation in schools and other organisations. In 1999, for example, the Inspectorate republished its indicators of strong and weak practice in the second edition of the document ‘Evaluating Schools’. This was followed by a complementary series of documents ‘Evaluating Subjects’, as well as ‘Evaluating Pastoral Care’; these papers again set out the indicators of strong and weak practice, this time at sub-ject level and in relation to Pastoral Care. Similar documents are available to support the con-cept of self-evaluation in organisations other than primary and post-primary schools. In further education and in training, for example, there are the ‘Improving Quality: Raising Standards’
materials, while comparable materials are currently being finalised for use in the pre-school and Youth sectors.
Periodic reports by the Inspectorate on specific issues in education and training also identify aspects of provision which need to be improved, and set out the characteristics of good prac-tice in a subject area, or in an area of provision (such as education of pupils in the sixth form), against which schools and other organisations can evaluate themselves. One example of a se-ries of documents, ‘Improving Subjects’, was published in 2001, and these leaflets summarise strengths and weaknesses in subject provision in post-primary schools, as well as priorities for action in relation to key areas for improvement. Again, this information is intended to help schools to reflect on and improve their own work.
In late March/early April 2003, the Inspectorate published, through a series of 14 conferences across NI, a set of materials entitled ‘Together Towards Improvement’. These materials were prepared in consultation with representatives from the Curriculum Advisory and Support Ser-vices (CASS) of the Education and Library Boards (ELBs), the Regional Training Unit (RTU), and the Council for Catholic Maintained Schools (CCMS), and, crucially, with serving principals from primary, post-primary and special schools across Northern Ireland. The materials have been designed to support schools with the process of self-evaluation leading to self-improvement. A key feature of the materials is the inclusion, for the first time in a published document, of a range of the Inspectorate’s quality indicators for schools.
In September/October 2003, the Inspectorate intends to publish a series of eight sector-based Digital Versatile Disks (DVDs) as further resources to help support those within the education and training sectors with the process of self-evaluation. Once again the materials have been developed in consultation with serving practitioners and with CASS, and provide guidance on self-evaluation in relation to pre-school, primary, post-primary, special education, youth work, alternative education provision, and vocational education and training.
The developing practices and the materials described above have served to bring about a shared language and understanding across NI of what constitutes good practice, and of the process of self-evaluation leading to self-improvement. Furthermore, they have served to dem-onstrate the importance of professional dialogue involving organisations, the Inspectorate and the support services in helping to promote lasting improvement in learning and teaching, and in the standards achieved by the learners. The long-term aim is that self-evaluation will become an integral part of all inspection activity, with the Inspectorate taking due cognisance of an organisation’s own agenda for improvement in the design and implementation of inspection. In this way, it is hoped that schools and others will increasingly regard inspection as an integral part of the process of continuous improvement, and not as an event to be endured for a period of time.
Educational Evaluation around the World 37
Objects
The Inspectorate inspects and reports on the quality of provision in individual organisations across the education and training sectors in NI. A wide range of organisations is inspected in-cluding:
pre-school centres;
nursery, primary, post-primary and special schools;
further education colleges;
training organisations;
the youth and community sector;
initial teacher education institutions;
the Curriculum Advisory and Support Services (CASS) of the Education and Library Boards (ELBs);
the NI Council for the Curriculum, Qualifications and Assessment (CCEA);
the educational support services, including educational welfare, the educational psychol-ogy services, and the peripatetic outreach services.
The materials described in the previous section, along with the structure and content of pub-lished reports and pre-inspection briefings by Reporting Inspectors, serve to guide organisations as to the nature, content, scope and purpose of inspection. This guidance has been enhanced significantly by the recent publication of the aforementioned ‘Together Towards Improvement’
materials, which provide a range of the Inspectorate’s quality indicators in relation to Ethos, Learning and Teaching, and Leadership and Management in schools. And, as indicated earlier, there are plans for the publication of similar materials for organisations other than schools, including pre-school centres and youth organisations.
The Inspectorate, in consultation with serving practitioners from across the education and train-ing sectors in NI, is prepartrain-ing to publish a Common Framework for Inspection in
Octo-ber/November 2003. The generic framework will provide information on the nature, content, scope and purpose of the range of types of inspection available, and include electronic links to further, phase-specific information to complement the generic guidance. The framework will also consider the inter-play of internal and external evaluation, and highlight the benefits of inspection to organisational improvement.
The content and nature of the inspection programme take account of the Citizens’ and Par-ents’ Charters, and of initiatives which the Government has introduced, for example, the School Improvement Programme, the New Deal Initiative and the Pre-school Expansion Pro-gramme. In addition, the inspection programme is designed or adjusted to provide evidence on issues about which the Departments and their Ministers require particular advice. In such cir-cumstances, surveys, involving visits to a representative sample of organisations, are often the main means of gathering the evidence required, resulting in the publication of a cross-cutting report; in these circumstances, separate reports on the individual organisations involved are not usually provided, although each organisation generally receives a short statement of the Inspec-torate’s findings. Each year, the scope of inspection activity is outlined in the InspecInspec-torate’s business plan.
Stakeholders
As indicated initially, the Inspectorate provides inspection services and information about the quality of education and training in NI for the Department of Education (DE), the Department of Culture, Arts and Leisure (DCAL), and the Department of Employment and Learning (DEL);
and the inspection programme is designed or adjusted in order to be able to provide evidence on issues about which the Departments or their Ministers require particular advice.
Evaluations are also provided for individual organisations via, for example, spoken reports to subject departments, senior management teams, and school governors; and to these groups,
Educational Evaluation around the World 38
the employing authorities, parents and the general public by way of written reports of inspec-tion and of follow-up inspecinspec-tions. In the primary phase, brief spoken reports are provided for each individual teacher at the end of inspection. In the case of cross-cutting reports, derived from survey work, the conclusions to the published reports make it clear that the evaluations are to help the organisations, the Departments and the support services establish a common agenda for improvement. The conclusions also indicate those with the lead responsibility for dealing with particular, identified areas for improvement. Within the last few years, the Inspec-torate has taken the lead in communicating the findings of cross-cutting reports directly to those most likely to effect change. Following a recent survey of provision for primary science and technology in a sample of primary schools across NI, for example, the main findings of the published report were relayed directly to the head teachers and science and technology co-ordinators in all primary schools in Northern Ireland. This was done through a series of semi-nars, organised jointly with specialist officers from CASS.
In June 2003, the now biannual review of education and training in NI by the Chief Inspector is due to be published. Prior to publication, the Chief Inspector will relay the main findings of the report to an invited audience which will include the relevant Ministers, the Permanent Secretar-ies of DE, DCAL and DEL, and key representatives from the various political partSecretar-ies, the employ-ing authorities and from business and commerce in Northern Ireland.
However, in keeping with its mission statement ‘Promoting Improvement (in the interest of all learners)’, the Inspectorate views the learners as the key beneficiaries of its evaluations across the education and training sectors.
Methods
As indicated earlier, the organisations to be inspected are identified by the Inspectorate and in recent years, schools and other organisations have become increasingly involved in determining the nature, scope, and timing of inspection. These important changes have been facilitated by the development of a range of types of inspection, in addition to the more traditional general and focused inspections. Furthermore, the increased range of types of inspection has increased the involvement of schools and other organisations in contributing to the assessment of the quality of their own provision. As with general and focused inspections, the findings of District Inspections, Two-part focused inspections, Quality Assurance Inspections, and Self-evaluative Follow-up inspections, are made public.
District Inspections are visits to schools, which are completed by the inspector who has the overview responsibility for a group of schools (the District Inspector). The visits focus on a single theme, e.g. Special Education Needs. A spoken report is given to the school by the inspector, and a short written report on the individual school is also produced. In addition, a report is published on the complete findings (i.e. the overview). The key purposes of the District Inspec-tions are:
to evaluate and report on a specific aspect of educational provision;
to further develop links among District Inspectors, head teachers and schools; and to encourage and enable schools to effectively monitor and evaluate aspects of their own provision.
As part of the Inspectorate’s commitment to its programme of review and continuous im-provement, all participating schools have been asked to evaluate the usefulness of the District Inspection. The responses have been positive. In particular, head teachers reported that the District Inspection helped them to evaluate aspects of their work, and they identified action which they had taken afterwards to effect further review and improvement of their work.
In the two-part focused inspection, one aspect of the inspection focuses on whole-school is-sues, including the school’s improvement work, and the second element focuses on a sample of subjects. In the case of the subject inspection, heads of department complete a
self-evaluation profile of the work in their department. The quality of the profiles has varied in line with the schools’ stages of development in reflecting critically on their own practice, but,
re-Educational Evaluation around the World 39
re-Educational Evaluation around the World 39