To clarify the quality assessment approach in the Netherlands, the case of the quality assess-ment of Physics and Astronomy programmes will be described, in order to show all the steps in the evaluation process.
The external quality assessment of the Physics and Astronomy discipline was planned for 2001/2002. It is the third time that the discipline has been assessed. It concerned the following programmes:
Physics Astronomy Applied Physics.
Nine universities were involved, among them the 3 Technical universities.
The first step was a letter to the board of the universities involved, signalling that the assess-ment of Physics and Astronomy was to commence in the near future. The letter enquired whether the programmes mentioned would participate, and if there were other programmes that should be included.
The faculty responsible for the programme(s) was invited to carry out a self-assessment, accord-ing to the rules of the protocol.1 The faculty had about 6 months to complete the
self-evaluation and to send the self-self-evaluation report to the VSNU.
At the same time, a letter was sent to the Board of Sciences at the VSNU, inviting the deans of the faculties to propose names for a chairperson. The proposal was checked by the VSNU and, as the proposed chairperson met the requirements, he was appointed by the chairman of the VSNU. Together with the chairman of the Board of Sciences, the chairperson of the external committee looked for potential members of the committee and proposed its composition to the VSNU. The department of quality assurance then checked the proposed members’ creden-tials against the criteria in the protocol. As a minimum, they should be independent, without conflicts of interest, have the necessary expertise and command authority among their col-leagues. An external committee normally has 5 to 7 members. There always has to be at least 2 experts from outside the Netherlands. In all cases, one member of the external committee is a student. In this case, the committee comprised 5 professional members and one student mem-ber. The chairman came from the USA, and another member from Belgium.
The department of External Quality Assessment of the VSNU appointed one of its employees as secretary of the committee. His task was twofold: to support the external committee as much as possible and to monitor the protocol and process.
1 VSNU, Protocol for the External Assessment of Educational Programmes 2000-2005. See www.vsnu.nl
Educational Evaluation around the World 168
When the members of the external committees were appointed, the composition was made known to the institutions to be assessed and to the Inspectorate. The secretary of the commit-tee started with scheduling the site visits.
In the mean time, the faculties were busy with the evaluation. They had to send the self-evaluation report to the VSNU at a certain date. The secretary of the committee checked the self-evaluation reports to see if they were complete and critical enough. In consultation with the members of the external committee, the secretary collected the final essays/master theses from the faculties and sent them to the members for assessment. At the same time the com-mittee members were asked to study the self-evaluation reports and assess their usefulness.
The members were also asked to formulate questions for additional information.
Prior to commencing the site visits, the committee members met for 2 days at the VSNU. The main aim of this meeting was to train the committee and to explain what was expected from of them. The time was also used for formulating the terms of reference, e.g. what did the committee expect as learning outcomes of a Master in Physics? What knowledge, skills and attitudes should be achieved after the study? It is important to have these terms of reference ready prior to the site visits, as these contain the domain specific, or subject specific, standards as seen by the external committee.
On the second day, the committee discussed the self-evaluation reports, shared out tasks and prepared questions for use during the site visit.
As mentioned, in the Netherlands, one and the same committee assesses all similar pro-grammes. In this case, the committee for physics had to carry out 9 site visits.
A site visit normally takes two-and-a-half days. It starts with a meeting of the committee to prepare itself for the visit. This is also the time to discuss and endorse the assessment report of the preceding visit. In the evening, there will normally be a dinner with the board of the univer-sity and the faculty. The next day there will be interviews with the writers of the self-evaluation report, the board of the faculty, staff, students and supporting staff. The committee has, more or less, the same programme the next day. It also looks at facilities, materials used, minutes of important committee meetings, etc. At the end of the two-and-a-half day visit, the committee will formulate its first findings and give a preliminary report to the faculty.
The secretary will produce a draft report to be discussed at the next visit. Following endorse-ment by the committee, the report will be sent to the faculty for comendorse-ment. It is up to the committee to decide what to do with the comments.
When all the visits are concluded, the committee will write the final report. This report not only includes the faculty reports, but also a comparative overview of all programmes being assessed and a statement of current practice of the discipline/subject at the time of evaluation. In the case of Physics, the report also contained some benchmarking with programmes in the USA and Europe.
The report will be sent again to the faculty to be checked for factual inaccuracies. After finish-ing the round of comments, the report is sent to the QA-department of the VSNU. A steerfinish-ing group will look at the report to see if it fulfils the requirements set out in the protocol. The steering group can ask the committee to be more complete and more precise, can accept or reject the report, but can never change the text of the report.
If the steering group accepts the report, the report will be printed and, on a predetermined day, the chairman of the external committee will hand over the report to the chairman of the VSNU in the presence of, in this case, the deans of the Science faculties.
Educational Evaluation around the World 169
The report will also be sent to the Inspectorate for the meta-evaluation. The meta-evaluation concerns two aspects:
Is this report of the committee an adequate and acceptable report? Was the committee independent? Was the necessary expertise present? Did the committee follow the rules of the game? Does the report cover all aspects? Are the judgements clearly supported by evidence?
The second question has more to do with the quality of the assessed programmes. The Inspectorate considers the report and, where there are serious shortcomings in the quality, will hand over yellow or red cards to the faculty, i.e. a warning that something has to be done. Any judgement concerning shortcomings is confidential.
In fact, the Netherlands already had an accreditation system, although this was informal: if you did not get a yellow or red card, you could consider yourself as accredited. However, there was no formal decision, and the criteria for the yellow and red cards were not always clear.
In the case of Physics, the general pattern of site visits and reporting has been followed. The report was published in 2002. It is waiting for the meta-evaluation.
Educational Evaluation around the World 170
Educational Evaluation around the World 171
New Zealand – Higher Education
John M. Jennings Director
New Zealand Universities Academic Audit Unit