John M. Jennings1 Director
New Zealand Universities Academic Audit Unit
Values and purposes
The purposes of evaluation within New Zealand higher education, for both quality assurance and quality development, are:
to protect the interests of learners
to ensure learners have access to opportunities for life-long learning to ensure available qualifications are meaningful and credible to assure learners that courses and programmes are well taught
to ensure qualifications are obtained in safe environments using appropriate teaching and assessment systems
to contribute to the enhancement of quality systems and processes that improve the qual-ity of research, teaching, learning and communqual-ity service.
The Education Act 1989 invests authority for accreditation, programme approval and institu-tional audit in the New Zealand Qualifications Authority for all institutions other than universi-ties, and in the New Zealand Vice-Chancellors Committee for universities.
The New Zealand Qualifications Authority is a statutory body set up by government as a result of the Education Act 1989. It has an overarching responsibility for quality assurance in secondary and tertiary education providers other than universities. Its Approvals, Accreditation and Audit section quality assures courses and qualifications offered by government and private training establishments and wananga (Maori tertiary providers), and degree and related qualifi-cations offered by polytechnics and colleges of education. The Qualifiqualifi-cations Authority has delegated quality assurance of sub-degree qualifications to the New Zealand Polytechnics Pro-grammes Committee (a committee of the Association of Polytechnics of New Zealand) in the case of polytechnics, and to the Colleges of Education Accreditation Committee (a Committee of the Association of Colleges of Education of New Zealand) in the case of independent teacher training institutions. From July 2003, the delegation to the New Zealand Polytechnics Pro-grammes Committee is to be extended to include Bachelor degrees.
The New Zealand Vice-Chancellors Committee is a statutory body which was given statu-tory status by the government in 1962, at the same time that the federal University of New Zealand was replaced by separate autonomous degree-granting institutions. With the Educa-tion Act 1989, the Committee assumed some of the funcEduca-tions of the former New Zealand Uni-versity Grants Committee at the interface between government and the universities, and was given the responsibility for quality assurance in all universities. To do this, the New Zealand Vice-Chancellors Committee has a standing committee - the Committee on University Aca-demic Programmes - which approves new programmes and significant changes to programmes and monitors their implementation. In 1993, the New Zealand Vice-Chancellors set up an inde-pendent evaluation agency - the New Zealand Universities Academic Audit Unit - to audit
uni-1 Acknowledgement is made of the assistance of Michael Steer and Peter Osborn, Approvals, Accreditation and Audit section, New Zealand Qualification Authority
Educational Evaluation around the World 128
versities' systems for monitoring and enhancing quality and to disseminate and commend good practice.
The authority for quality assurance in New Zealand higher education
The normal terminology used in New Zealand, and generally accepted in the education sector is accreditation, approval and audit.
Evaluation would be understood to be audit of systems for monitoring academic quality, to assure the quality of core activities (research, teaching, learning, community service) and to assist in the enhancement of those activities.
Quality development would be understood to be the identification, development and im-plementation of institutional systems and processes to enhance the quality of an institu-tion's core activities.
Quality assurance would be understood to incorporate accreditation, approval and audit, where accreditation is understood to refer to the accreditation and registration of provid-ers to deliver approved programmes and courses of study leading to qualifications,
ap-New Zealand Qualifications
Educational Evaluation around the World 129
proval refers to the approval of the programmes and courses leading to qualifications, and audit refers to the evaluation of the effectiveness of institutional systems for monitoring and enhancing academic quality.
Objects
Before government subsidy can be given to providers of higher education who seek to offer approved courses of study leading to qualifications, the providers themselves must be regis-tered, the courses they wish to offer must be approved, and the providers must be accredited to deliver them.
The New Zealand Qualifications Authority approves the courses offered by non-university pro-viders and accredits the organisation to deliver them. In 2001, there were 490 non-university institutions delivering government subsidised courses and programmes to 161,803 students:2
Institution type Number of
institu-tions
Student Enrolment Tertiary Education Institutions (public):
polytechnics 21 87,965
colleges of education 4 10,894
wananga (Maori tertiary providers) 3 11,278
Private Training Establishments (private):
institutions receiving government subsidy 462 51,666
A large number (over 450) of private training establishments offer courses that are not directly funded by the government.
The Committee on University Academic Programmes of the New Zealand Vice-Chancellors Committee approves all new programmes of study in universities leading to new qualifications and all major changes to programmes of study in existing qualifications; the New Zealand Uni-versities Academic Audit Unit evaluates the quality assurance processes in all uniUni-versities includ-ing the processes associated with the institutional approval of minor changes to programmes of study.
Institution type Number of
institu-tions
Student Enrolment Tertiary Education Institutions (public):
universities 8 125 668
With respect to the ongoing monitoring of the quality of degree programme delivery and main-tenance of standards in non-university institutions, the New Zealand Qualifications Authority appoints a Monitor nominated by the approval panel for each new qualification as each has come on stream from 1990. Each Monitor conducts a desktop paper audit and site inspection to ensure the institution implements fairly and adequately the quality management systems and assessment moderation procedures that had been agreed to at the time of the approval of the qualification.
In 2000, after almost ten years of operation in quality assurance, the Qualifications Authority supplemented what it described as a ‘front-end check of a provider’s policies and their capacity to deliver what they claim they will deliver’3 with an external quality audit process. This requires self-evaluation by providers, takes account of the advances made by providers in quality
assur-2 Statistics taken from Ministry of Education, New Zealand’s tertiary education sector: profile and trends 2001, Wellington, Ministry of Education, 2002.
3 OnQ, [New Zealand Qualifications Authority] June 2000, p.2
Educational Evaluation around the World 130
ance systems, recognises the increasing diversity among providers, and gives providers a greater ownership of the process. With over 900 private training establishments, the audit process was planned to be phased in over a three year period and, at the present time, almost all providers have been audited at least once, and some, where concerns were identified, as many as four times. The ongoing review of over 1100 quality audits has informed the revision of the Quality Assurance Standard for Private Training Establishments, Government Training Establishments and Wananga to enhance its effectiveness.
The most common response of providers to the audit process has been to redevelop their own quality assurance processes so that they are able to more clearly demonstrate that the organisa-tion is meeting the needs of learners and other stakeholders, and is complying with the specific requirements of the Quality Assurance Standard. In general, at this early stage of the imple-mentation of the audit process, organisations have welcomed the opportunity to carry out this revision, and report that they are in a better position to ensure the quality of their operations.
With respect to the ongoing monitoring of the quality of programme delivery and maintenance of standards in universities, the New Zealand Universities Academic Audit Unit has completed two cycles of audit. Cycle 1 audits, in 1995-1998, were fully institutional, auditing systems and processes associated with all aspects of an institution's activities. Arising from a review of Cycle 1 audits, it was agreed that Cycle 2 audits would be the first of three theme audits that were to be smaller-scale tightly prescribed audits. The first of the three themes was to be on research matters, the second on staff matters, and the third on academic programmes. Cycle 2 audits - on the national theme of research policy and management, support for research students and the research-teaching nexus, and on a topic nominated by the institution - were carried out within a tighter timeframe of 2000 and 2001. In spite of the intention, the university workload of Cycle 2 was as great as the fully institutional audit of Cycle 1, and the theme concept has been discontinued for the present. Cycle 3 audits, which begin in 2003, will have a focus on teaching quality, the delivery of programmes, and the achievement of learning outcomes, and will be carried out over four years.
Stakeholders
Evaluation is used as a mechanism to assist institutions in their review of quality and to assure stakeholders - particularly students but also government agencies, the professions, business and industry - of the maintenance of quality in the core activities of research, teaching, learning and community service. Evaluation is also designed to contribute to the enhancement of these core activities and to the learning environment in which these activities take place, in order to better equip graduates to add value to society. Evaluations, therefore, are directed primarily at the institution's management, academic staff and students, and the evaluation processes are designed to involve those sectors as well as graduates, the professions, business and industry.
The primary indicator that these reports are written for the institutions themselves is the inclu-sion - and clear presentation - of requirements and recommendations for improvement. The procedures require follow-up by the institution to the audit agency; while not regarded as pub-lic, the Official Information Act 1982 would require an institution to provide a copy of a follow-up report on request from a member of the public.
Audit reports on non-university providers prepared by the New Zealand Qualifications Authority have not been released as public documents, but are available on request to individuals who have valid reasons for requesting them. Individuals requesting a report are referred, in the first instance, to the provider concerned. Where the provider refuses the request, the Qualifications Authority will make a copy available.
Audit reports on universities prepared by the New Zealand Universities Academic Audit Unit are released as public documents and, therefore, they are available to anyone for consideration, and aspects of the reports are often reported by the media. Copies of the reports are sent to a wide range of stakeholders, in particular to government educational and audit agencies.
Educational Evaluation around the World 131
Methods
The New Zealand Qualifications Authority’s Approvals, Accreditation and Audit section con-ducts evaluations:
at the time an organisation is registered
each time an organisation applies for course approval and accreditation
at the time of scheduled quality audits that occur at intervals of between one and three years or more regularly if the Authority is concerned about aspects of the institution's sys-tems and/or quality of teaching and learning environment
at the time of monitoring visits for degrees and related qualifications.
The New Zealand Polytechnic Programmes Committee conducts evaluations:
each time a polytechnic applies for course approval and accreditation beyond its current scope of accreditation
at the time of scheduled quality audits that are conducted at intervals of up to four years.
The Colleges of Education Accreditation Committee conducts evaluations:
each time a College of Education applies for course approval and accreditation at the time of monitoring visits.
The New Zealand Universities Academic Audit Unit evaluations of all universities are conducted according to a timetable agreed to by the universities.
The criteria against which institutions are evaluated for accreditation, and against which courses are evaluated for approval, have been determined by the New Zealand Qualifications Authority in consultation with the New Zealand Vice-Chancellors Committee and have been published by the Government in The New Zealand Gazette. The criteria are reviewed from time to time. The quality assurance agencies that undertake the work - the New Zealand tions Authority (and any quality assurance agencies with delegated powers from the Qualifica-tions Authority) and the universities' Committee on University Academic Programmes – have developed their own procedures to operationalise the criteria in ways best suited to the institu-tions they evaluate. It is the responsibility of quality assurance agencies to ensure proper im-plementation of the criteria.
The audit evaluations of institutions’ quality assurance and quality enhancement systems are carried out by the New Zealand Qualifications Authority and the New Zealand Universities Aca-demic Audit; each has developed its own acaAca-demic audit processes.
Assessments in the evaluations are undertaken by independent panels appointed by the quality assurance agencies. The panels are responsible for any requirements and/or recommendations that arise from the process. Evaluation panels are of various types depending on purpose.
The Approvals, Accreditation and Audit section of the New Zealand Qualifications Authority has responsibility for the evaluation of providers seeking registration, course approval or accredita-tion.
Applications for accreditation as a registered provider are evaluated by one of 17 trained and qualified Quality Auditors who assess the application against a clearly defined stan-dard – the Quality Assurance Stanstan-dard for Private Training Establishments, Government Training Establishments and Wananga.
Applications for course approval and accreditation are evaluated by a panel consisting of a Quality Auditor and specialists in the subject area(s) of the course. There is no prescribed number of panellists for non-degree courses.
Educational Evaluation around the World 132
Applications for degree qualifications are evaluated by a panel of eight members, compris-ing:
two representatives from the universities, two industry representatives,
a representative from another tertiary provider offering a similar course, a representative from the applying organisation,
a panel chair,
a quality systems evaluator.
The New Zealand Qualifications Authority’s Quality Auditors conduct audits of providers against the criteria defined in the Quality Assurance Standard for Private Training Establishments, Gov-ernment Training Establishments and Wananga. Most commonly, only one Quality Auditor is involved, but larger providers may require assisting auditors.
The New Zealand Polytechnic Programmes Committee - under delegation from the New Zea-land Qualifications Authority - conducts the quality audits of polytechnics with a panel of four, comprising:
a New Zealand Polytechnic Programmes Committee Lead Auditor, a representative from the polytechnics,
a representative from the industrial sector,
a representative from the New Zealand Qualifications Authority (where the polytechnic offers degrees).
The New Zealand Universities Academic Audit Unit evaluation panels comprise:
two or three New Zealand academics,
one New Zealand-resident non-academic (from business or industry),
one academic or former academic from outside of New Zealand (usually from Australia), the Director of the Unit.
Evaluations of accreditation of providers and approval of new courses are not made public;
audit reports, on the other hand, are, or soon will be, public documents in all cases.
The institutions that have been audited are required to present follow-up reports to the quality assurance agency in accordance with an agreed timetable. The New Zealand Qualifications Authority sets down requirements as well as recommendations. Requirements must be satisfied before a new programme is approved, and requirements in an audit must be satisfied by a certain date if accreditation and registration are to continue.
Evaluations for accreditation and approval of courses are undertaken by quality agencies which develop their own procedures to operationalise criteria as published in The New Zealand Ga-zette. The current criteria, gazetted in December 2002, are:
Title, aims, learning outcomes and coherence. The adequacy and appropriateness of the title, aims, stated learning outcomes and coherence of the whole course.
Delivery and learning methods. The adequacy and appropriateness of delivery and learning methods, for all modes of delivery, given the stated learning outcomes.
Assessment. The adequacy of the means of ensuring that assessment procedures are fair, valid, consistent and appropriate, given the stated learning outcomes.
Acceptability of the course. The acceptability of the proposed course to the relevant aca-demic, industrial, professional and other communities, in terms of its stated aims and learning outcomes, nomenclature, content and structure.
Regulations. The adequacy and appropriateness of the regulations that specify require-ments for admission, credit for previous study, recognition of prior learning, course length and structure, integration of practical/work-based components, assessment procedures, and normal progression within a course.
Educational Evaluation around the World 133
Resources. The capacity of the organisation to support sustained delivery of the course, in all delivery modes, with regard to appropriate academic staffing, teaching facilities, physi-cal resources and support services.
Evaluation and review. The adequacy and effectiveness of the provision for evaluation and review of courses for monitoring the ongoing relevance of learning outcomes, course de-livery and course standards; for reviewing course regulations and content; for monitoring improvement following evaluation and review; and for determining whether the course shall continue to be offered.
Research (for degrees and postgraduate qualifications). The adequacy of provision of re-search facilities and support of staff involved in rere-search; the levels of rere-search activity of staff involved in the course and of ways in which the research-teaching links are made in the curriculum.
In the case of audit evaluations, the New Zealand Qualifications Authority evaluates against its own Quality Assurance Standard for Private Training Establishments, Government Training Es-tablishments and Wananga, which requires consideration of the quality of the following:
Goals and objectives - measurable goals and objectives for education and training, Systems to achieve goals and objectives - the implementation of quality management systems to achieve goals and objectives by showing evidence of:
adequate and appropriate governance and management, the recruitment, management and development of its people,
the provision of adequate and appropriate physical and learning resources, the supply of adequate and appropriate information, entry and support services to learners,
the adequate and appropriate design, development, delivery and review of its educa-tion and training programmes,
the adequate and appropriate systems of assessment and moderation for assessing learners,
adequate and appropriate reports on learner achievement,
adequate and appropriate means of research where degree programmes are offered The achievement of goals and objectives - evidence that it is achieving its goals and objec-tives and can provide assurance that it will continue to do so.
The Quality Assurance Standard for Private Training Establishments, Government Training Es-tablishments and Wananga was first approved by the New Zealand Qualifications Authority Board in October 2000 and has undergone two revisions since then.
The New Zealand Universities Academic Audit Unit has evolved its own audit manual through three editions between 1995 and 1998, requiring consideration of the quality of the following:
structure and planning, quality management system, the Treaty of Waitangi, staff matters,
courses and programmes,
teaching, learning and assessment, research and teaching,
reviews,
feedback processes,
joint, franchised and external programmes, support for students,
facilities and resources.
Educational Evaluation around the World 134
For the Cycle 2 theme audit on a national theme of research policy and management, support for research students, the research-teaching nexus, and institutional themes agreed to by the Unit, a supplement to the manual was produced detailing detailed expectations with respect to
For the Cycle 2 theme audit on a national theme of research policy and management, support for research students, the research-teaching nexus, and institutional themes agreed to by the Unit, a supplement to the manual was produced detailing detailed expectations with respect to