Johan van Bruggen
Staff Inspector for International Co-operation
Central Office Inspectorate of Education, the Netherlands
Values and purposes
As in all other developed countries, we ask the three basic evaluative questions at the different levels of education:
The level of the individual pupil, student, teacher;
The level of the individual school;
The level of the educational system at local, regional or national level.
At the individual level
At the individual level, several questions arise: How good is the student in a particular subject?
What improvements have been made over the last few weeks? Is that progress good enough for the student to pass to the next course?
Teachers play a dominant role in these evaluations, sometimes assisted by tests or examinations run by schools, or nationally (see paragraphs below about objects and method). It is clear why these evaluations are important: the decision of pass/fail, and thus about the future of students rest upon them. At this level, the responsibility for evaluation in our country lies in the hands of teachers, who evaluate their students, either alone or as a team.
The inspectorate has no task here, except to assess in an evaluation of a school, whether the function of evaluation of student progress is being fulfilled well enough (i.e. regularly, system-atic, honest, complete, with methods that are as objective as those requested by the issue, and other indicators). For some evaluations of decisions about students, the laws concerning schools contain prescriptions, which are often of a general character (e.g. “the school has to have a well-developed policy for assessing the progress of students through the grades”), sometimes more specific (e.g. concerning the organisation and content of school bound ex-aminations for certain subjects at the end of senior secondary schools; and the function of the work of the national institute for assessment and examinations CITO in that process).
It should be clear why the prescriptions for schools, and the checking and evaluating tasks of the inspectorate, are considered important: for the protection of the rights of individual stu-dents to fair, professional and comparable decision-making about key stages in their develop-ment and about their educational chances; and for guaranteeing society (employers, next schools, authorities) that diplomas, certificates, grades are valid and reliable, and have compa-rable value across the country and over time. So, here, a general function of the evaluative work of the inspectorate of education is seen as: guaranteeing a bottomline of quality by the inspection of provisions and actual practice in schools and reporting findings to schools, au-thorities and the public.
In addition, evaluation of the quality of the individual teacher (or principal) is important. Is he/she good enough to receive a new contract? Is he/she good enough to be asked to run this difficult class, or course? Also, here the decision-orientation is clear, guaranteeing adequate quality at this very important input-side.
As mentioned, decisions about these issues are in the hands of the school managers and school boards who evaluate teachers and principals. The inspectorate has to evaluate how these
par-Educational Evaluation around the World 74
ties fulfil their tasks: an aspect of quality is if the “personnel policy” in a school is well devel-oped; and one of the issues, then, is if the evaluation of teachers and principals is done ade-quately. The inspectorate has no function in individual decisions about appointments, promo-tion or dismissal of staff.
So, both “aspects of quality “ of a school (a good system of assessment of the progress of students; a good system of evaluation of staff) are evaluated by the inspectorate in its assess-ment of “the quality of a school”.
The institutional level: school, faculty, learning-center, etc.
At this second level of evaluation – the school – more and more people believe that, indeed, 'school matters' (P. Mortimore and others: 'School matters: The Junior years', Open Books, Wells, 1988), and that there are important differences in quality among schools or faculties. I refer to the literature about effective schools, e.g. Reynolds, Scheerens, and many others.
Evaluation has to uncover these differences. This is the background for the rapid development of public self-evaluations carried out by schools themselves, sometimes, as in the Netherlands, Northern Ireland, Scotland and other countries, under a legal obligation. Such an internal evaluation can be done rather informally, or can be only very partial, e.g. by publishing the results of the school in national examinations and comparing these with national averages. The internal evaluation can also be rather formal, with the involvement of a “critical friend”, i.e. a form of committee.
The conviction that “school matters” is also the background for the frequent public reports of external evaluators: inspectors of schools, audit-committees for faculties or universities, etc.
Also this type of evaluation is done in the Netherlands. Two aspects of quality have already been mentioned that are evaluated in such an external inspection, but, of course, there are more, e.g. the learning results, the pedagogical climate in the classrooms, and other ones. See further when we return to the objects and method of evaluations, which forms the core of the work of the inspectorate. A question immediately arises regarding the evaluation of the learn-ing results: are these assessed in an absolute way, or in a relative way with a leadlearn-ing question and are these learning results good enough taking into account the intake of students and their socio-economic backgrounds? For the “pedagogical climate” important evaluation ques-tions, for example, are: is it safe, warm, challenging, etc.? We shall return to the objects and method of external inspection later, as said, but the examples of questions illustrate the impor-tant question. But, firstly, something about values and purposes.
The first ‘decision-oriented function” of the evaluative information about the quality of a school is to help the parents (or elder students in senior secondary schools, further and higher education) in their decisions about school-choice and in their involvement in parent committees and other groups involved in schools.
Parents will often ask, “Is this the best school for my child?” To find the answer, they can use public (printed or Internet) reports of the inspectorate about schools, or they can use articles in regional newspapers, where journalists may use the same original reports plus their own sources. Some regional newspapers publish ranking-lists of schools. Students will sometimes make selections of universities using reports of evaluation-committees or ranking-lists published in journals e.g. Focus in Germany; Times in the USA; Le Monde in France; Elsevier or Trouw in the Netherlands.
Of course, there is also a lot of informal “evaluation”, done by people at in “over-the-fence”
discussions with their neighbours about the differences between two or three primary schools in the neighbourhood; or by young people of 16/17 who ask each other at the beach or a con-cert about the best faculties in the best universities in the best towns.
An important fact in our country – differing from most other countries – is that there is full freedom of school-choice: parents or elder students are fully free to choose between schools,
Educational Evaluation around the World 75
e.g. publicly or privately governed, Protestant or Islamic, etc. So, evaluative information about schools can have its place in this selection process. It is known that, in most European coun-tries, school choice is also becoming freer, but not as radically as here.
The second ‘decision-oriented function” of the evaluative information – internal self-evaluation or external by the inspectorate about the quality of an individual school, programme or course within a school – is focused on the stimulation of the autonomy of the individual school. This autonomy of a founding board, association or group of parents, to choose a mission, an iden-tity (religious, pedagogical, didactical, organisational), textbooks, teachers, accents in the cur-riculum, etc., is deeply rooted in the history of Dutch educational policy.
One expression of the desired and promoted autonomy of schools in matters of pedagogy, profile, curriculum (partially free) and organisation is that schools are obliged to make their own school plan and school programme. In these documents, schools have to express their identity (religious, pedagogical, organisational, etc.) and to inform themselves and interested parties what this identity means in terms of practical consequences. The belief – like in most Western countries - is that this obligation to develop a school plan and school programme stimulates the choice – by the schools – of a sharper profile and, thereby, more variety between schools, leading to better quality. In this process, the self-evaluation is also an important – obligatory – step. Put simply, as a school, you cannot develop ideas about new development and improvement if you do not know your own reality, and do not have your own evaluation of that reality. The title of the well-known Scottish framework for self-evaluation and external inspection perfectly expresses this idea: “How good is our school?” External inspection has its place in between, i.e. between the autonomy of a school (and its expression through the re-sponsibility of the school for self-evaluation, and the consequences that the school connects to it) and the desire of society to provide for an external, independent, professional, public evalua-tion of school quality. This external evaluaevalua-tion, on one side, funcevalua-tions as a guarantee for suffi-cient “basic quality” but also as a stimulus, a mirror, for the school to develop and to bench-mark itself against such an external assessment. Of course, there are tensions here, which we will return to.
We can thus see that the second function of the evaluative information about individual schools (stimulation of autonomy, choice and variety) is connected inseparably with the third function of this same evaluation about the quality of an individual school: to inform authorities (school boards, regional authorities, the minister of education) about the quality of individual schools in order to enable them to take measures, in particular, if this quality is below expected or prescribed standards. We call this the guaranteeing function of the inspectorate.
The system level
At the system level (national, regional, county, provincial, town), the inspectorate also evaluates the quality of education.
Here questions are more general:
How good is our system compared with other systems, e.g. average learning results for mathematics (TIMSS, PISA)?
How effective is the use of ICT skills by students of a certain age?
How good are our schools and our teachers and our pupils in general?
Or – a very different issue - is the system flexible enough to allow pupils to take their own, best route through its opportunities?
Does it provide enough innovation-stimuli?
Is it economic?
Does it provide value for money?
The inspectorate investigates these types of questions by aggregating the results of the many (ca. 2300 inspections of primary schools, 300 of secondary schools, 15 of the large further education institutions) school inspections per year, and by using results of educational research.
Educational Evaluation around the World 76
An annual report about the state of education is sent to parliament – through the minister who, by law, has no right to change it but only to give his comments – and this raises much interest and discussion. The inspectorate, and research groups occasionally cooperating with the inspectorate, also deliver thematic reports on certain issues, questions or topics, based on thematic inspections in a sample of schools.
Why is this evaluation at system level important? Because it delivers evaluative information that can contribute to the national public debate about what “we” find important in our schools and about what “we” feel should be improved or changed. So, this evaluative information is helping to set the agenda for policy-making. Of course there is no direct link between informa-tion and policy-making, as we know well enough from research about this difficult issue, but it is also known that this information has its place, depending also on its quality, timing, and the standing of the inspectorate. There are many questions and issues here, but I must leave these aside.
Here is a summary of my views about why, in our country, evaluative information at the three levels is found to be important:
a. in order to serve parents and students in their decision making regarding school choices and involvement in school policy and school improvement;
b. in order to help – through providing them with an external, professional, independent evaluation beyond the obligatory self-evaluation – school boards, principals, teachers, con-sulting and advisory centres to fulfil their responsibilities to better formulate their intentions (mission, aims, content, climate, etc.), and to better work towards there realisation, always within the boundaries of what has been laid down in educational laws regarding aims, content and organisation, etc.
c. in order to guarantee that responsible authorities are able to know whether individual schools and the system as a whole are good enough, i.e. meeting agreed standards; deliv-ering quality comparable with schools in comparable circumstances; serving society’s needs well enough; at internationally comparable standards; etc. It is not the inspectorate that has to take action if the evaluation shows that “it” (a school, a theme, the system) is not good enough. The inspectorate is responsible for delivering the evaluative information timely, and so serves as an early warner and as a diagnostician.
Writing about the objects in evaluative work, we will give more details about these three basic perspectives.
Objects
At the student level
The evaluation of achievement and attainment of students, as carried out by teachers, groups of teachers, examination committees, or with the help of tests or observation-instruments pro-duced by the National Institute for Assessment and Examinations (CITO) is, of course, a very important part of daily formative and summative evaluation. It is, however, done in the Nether-lands in the same way as in most other European countries. It is worth remarking that increas-ingly in Dutch schools a “culture of more objective assessment” is developing. Teachers use more and more the tests or instruments that are delivered by textbook-series (formative evalua-tion) or CITO, or by regional advisory-centres. Groups of teachers even develop their own more objective instruments of assessing the progress of students. More than 85% of primary schools use the well-known CITO-test “End of Primary School” for mathematics, language skills and information skills in order to complete their advice per student for the best track into the sec-ondary school system. The system of combined school-bound and national examinations con-cluding secondary school, each providing 50% of the final grades, is not seriously in question.
Of course, improvement of these aspects of evaluation in training, in-service-training, material delivery, etc. is a continuous task, as is the assurance of the quality of examinations, achieved by inspection.
Examinations with a central, standardised grade is not only a matter of simple paper and pencil tests with multiple choice items – although for reasons of objectivity, cost, speed of correction, that type is frequent in most of our national examinations. 'Free composition' by students on
Educational Evaluation around the World 77
the basis of centrally developed open questions, dossiers (cases) or films (videos, tapes, CD ROM, etc.) is happening and allows longer answers, drawings, essays, constructions, etc. These are judged by external experts and/or by teachers from other schools who receive these 'free compositions' for grading along with clear instructions about aspects and criteria, without knowing the names of students and schools.
Inspection of the examinations – as set nationally and by schools – is focused on the following issues:
Do the examinations cover the vital parts of the agreed national/school objectives and national/school content matter?
Do they have the ´right level` of difficulty?
Do they have enough integrity, including fraud prevention (no secret questions; no mobile telephones; no 'friend' as a marker; etc.)
Each year the inspectorate examines a sample of schools (further and secondary) and a sample of subject examinations regarding the above aspects.
When we evaluate in a school, if the routine formative and summative assessment of the pro-gress of pupils is done well enough, we look into issues such as: the use of textbook-bound-tests; the administration of, for example, “follow-me-systems”; the learning paths of pupils;
pupil dossiers and portfolios. We talk with teachers and students about this work, whether it is fair, comprehensive and well linked with diagnostic and improving actions?
At the school level
Before I can go into the real details of “what is evaluated” by schools that self-evaluate, and inspectors who carry out an external evaluation, it is important to describe the recent legal and political background to this work. This background may be found in the recent Supervision Act.
On 5 June 2001, the Dutch government proposed the ‘Wet op het Onderwijstoezicht’, Supervi-sion Act (Dutch acronym WOT) by sending it to Parliament. This was the concluSupervi-sion of a period of three years of “white papers” and rather intensive public and parliamentary debates. In these debates, various aspects of the tensions, mentioned earlier, between the desire to pro-mote more autonomy, profile and variety among schools on one side, and to guarantee mini-mum quality and to promote outstanding quality though external inspection on the other side were discussed in detail, and brought to workable solutions and “checks and balances” to the systems. In sessions in October and December 2001 and June 2002, the two chambers of par-liament proposed amendments, and discussed these with the government. The Law came into force in September 2002, and the new inspection-process, based on this first law on inspection in the 200 years’ history of the Dutch Inspectorate of Education, commenced in January 2003.
This Law provides a legal basis for independent, professional and public evaluations of the qual-ity of schools and the system by the Inspectorate of Education. This type of whole – school – evaluation has been developed in the Netherlands beginning with experiments with 180 schools in primary education in 1991-1993; there is much similarity between this type of in-spection work and that found in many other countries in Europe, and in Hong Kong, New Zea-land etc. Since 1995, the Standing International Conference of Central and General Inspector-ates of Education SICI has been the association of, now, 20 European inspectorInspector-ates of educa-tion. Most of these bring into practice this type of external inspeceduca-tion. There are various work-shops, reports and other publications of SICI that provide lists of quality indicators, descriptions of frameworks for inspection, guidance on dealing with schools with quality problems, etc. – for more information, see www.sici.org.uk
With the Supervision Act (WOT), the Dutch government intends to ‘strengthen the position of the Inspectorate of Education as a provider of independent judgement’. The Supervision Act is important because schools are, on one hand, given more and more autonomy and responsibil-ity for their own qualresponsibil-ity, with fewer rules and regulations on the input side (curriculum,
With the Supervision Act (WOT), the Dutch government intends to ‘strengthen the position of the Inspectorate of Education as a provider of independent judgement’. The Supervision Act is important because schools are, on one hand, given more and more autonomy and responsibil-ity for their own qualresponsibil-ity, with fewer rules and regulations on the input side (curriculum,