The CEEC represents a very special case in Canada. It is the only agency with the mandate to evaluate non-degree granting institutions, both public and private. This mandate covers the evaluation of the institution itself, its program and its academic policies. The CEEC is an autonomous agency created by an Act of the Government of Quebec. It is composed of three commissioners who are assisted by a staff of civil servants. All college level institutions in Que-bec are subject to the evaluation of the CEEC.
The CEEC was created in 1993 in the wake of a major reform of college education in Quebec.
Before the reform, the college system was highly centralized, with programs completely elabo-rated by the Ministry of Education. Colleges had little autonomy to adapt these programs to their environment and their internal context. One of the major aims of the reform was to de-centralize the system and give more responsibilities to institutions, particularly with respect to the educational programs, in order that they be more responsive to the needs of their commu-nity and to the constraints of their environment. The CEEC was created in this context as a mean for the colleges to account for their realizations and for the government to make sure that the decentralization proceeded in an ordered manner.
Values and purposes
At the beginning of its operation, the CEEC clearly indicated that it would fulfil its mandate with the objectives of improving the quality of college education, testifying to the value of the edu-cation offered in each institution, and helping the institutions in the development of a culture of evaluation. This last objective came from the fact that few colleges had experience in the field of quality assurance and program evaluation. These objectives clearly define the CEEC as a quality improvement centered agency.
The first objective was pursued in all the evaluations realized by the CEEC, whether they were evaluations of program or later evaluations of institution. The Commission is supportive of the institutions in their evaluations and reports and leaves them space and time for improving their performance. The Commission’s reports point out strengths and weaknesses and include ob-servations and recommendations. Recommendations call for appropriate actions to correct the situation. The reports are public documents, posted on the CEEC’s website.
In cases where a program or an institution has deteriorated to a point where there is little hope of progress, the Commission recommends that it be closed. In this sense, the CEEC is also a quality assurance agency. This recommendation is made to the Minister of Education. It is sel-dom used.
The fact that the CEEC’s reports are public documents increases considerably their efficiency. On many occasions, institutions have used them to bring about changes that would otherwise
Educational Evaluation around the World 156
have been difficult to realize. In this sense, the CEEC is seen by many colleges as an efficient agent of change.
As indicated above, at the time of the reform and the creation of the CEEC, colleges had little experience in the field of evaluation. For this reason, the Commission developed detailed guides for program evaluation, gave workshops to help institutions to realize their
self-evaluations and selected and trained experts from the college community. In this way, colleges have learned how to realize efficient program evaluations. A recent audit of the quality assur-ance processes of the colleges has shown the importassur-ance of the progress made in the devel-opment of a culture of evaluation and the realization of efficient evaluations.
Objects
The initial mandate of the CEEC was to evaluate for each college, public or private:
the institutional policies on the evaluation of student achievement the institutional policies on program evaluation
the programs of study.
This mandate was recently extended to include the evaluation of the institutions themselves with regard to their academic and organizational management, their strategic plans and their plans to increase their students’ success rate.
When evaluating a program, the Commission looks at the following: its relevance with regard to the needs of the society and their students; the coherence of the curriculum taking into account its objectives; the pedagogical methods utilized; the competence of the teachers; the adequacy of the resources; the assessment of the achievement of the students; and finally, at the program management.
When evaluating an institution, the CEEC examines the relevance and clarity of its mission and objectives, the efficiency of its organization, the adequacy of its resources, its planning process, its integrity and its outcomes. The Commission audits separately the quality assurance mecha-nisms developed in its policy on program evaluation.
The CEEC began its operation by the evaluation of programs and chose programs that were widely offered throughout the province, like computer technology and social science. In this manner, the Commission hoped to diffuse good evaluation practices in the institutions and, at the same time, give a first assessment of the quality of college education. After evaluating a number of programs in each institution, the Commission decided to audit the internal program evaluation practices of each institution. Recently, it began evaluating individual institutions in terms of organization and management. The Commission felt that this last operation would help institutions to better identify the origin of problems that appeared in a recurrent manner in the evaluation of programs. The CEEC also believed that institutional evaluations would per-mit the measurement of the extent to which the institutions had assumed the new responsibili-ties they were given by the 1993 reform.
Stakeholders
It should be clear from the above that the institutions are the first and main beneficiaries of the work of the CEEC. Though many stakeholders take advantage of the improvement in the quality of college education – in the first place the students, but also the employers, the universities when they receive college graduates and the society in general –, it is the institutions them-selves that directly benefit from the evaluations of the CEEC. It is their activities, which are evaluated, their educational programs, which are strengthened, and their reputation which is enhanced. It is to them that the evaluation reports are addressed and for them that they are prepared, even though they are eventually made public and available on the website of the
Educational Evaluation around the World 157 CEEC. Finally, the CEEC adopted evaluation procedures and used instruments that were specifi-cally designed to help colleges to develop their own culture of evaluation.
Methods
As indicated before, the CEEC is autonomous. It chooses the institutions and programs to be subjected to a particular evaluation, the evaluation methods and the moment of the evaluation.
It is fully responsible for its judgments and its reports.
In all its evaluations, the Commission uses similar procedures. It first develops the necessary instruments, guides, questionnaire samples with the help of an advisory committee comprising academics and professionals from the labor market. In the case of the evaluation of a program, the institutions offering the selected program are then requested to carry out their
evaluation following the criteria and suggestions of the guide. Upon receiving the
self-evaluation reports, the Commission organizes site visits. The visiting committees are composed of one of the commissioners and three or four experts of the field. Each committee is chaired by the commissioner, and a professional of the CEEC acts as secretary. The report of the visiting committee constitutes the basis for the Commission’s report. This final report is adopted by the three commissioners during one of the CEEC’s formal meetings. It is then first sent to the institu-tion for their reacinstitu-tions. Modificainstitu-tions can be made before a definite report is adopted. In other type of evaluation – audit and institutional review –, the same procedure is followed.
For its program evaluations, the CEEC has developed a general guide in which are presented its criteria. This general guide serves as the basis for the elaboration of more specific guides adapted to the situation of each program. In these specific guides, the CEEC indicates which aspects of the program should be examined more closely depending on the particular context, for instance, actualization and equipment in computer sciences, coherence in social sciences.
For each criterion, the specific guides precise what type of information should be collected and make suggestions with regard to the analysis. The CEEC expects that the institution will have identified its weaknesses in its self-analysis and adopted a plan of action for improvement.
Special guides were developed for the audits and institutional evaluations. Of course, the crite-ria are quite different, but the approach is similar. There is, however, a major difference be-tween these three types of evaluations: the framework of the evaluation.
In the case of a program evaluation, the objectives of the program constitute the reference against which the evaluation must be done. These objectives must be, and generally are, pre-sented as competences that the graduates should have acquired. Therefore, the evaluation looks at the process and resources, as well as at the outcomes. Information related to out-comes, such as employer and graduate surveys, employment rates, must be used as well as internal data on courses, graduation rates, student satisfaction, faculty and resources.
When the CEEC is carrying out an institutional evaluation, it is the mission and objectives of the institution that serve as references. Here the focus is on the organization of the institution un-der review, to determine if it is optimally adapted to its mission and objectives. The outcomes are obviously looked at, but they are of a more general nature and may depend very much on the type and mission of the institution, e.g. small private institution, urban college, regional institution, etc.
As for the audits, the institutional policy on program evaluation constitutes the reference, and the audit is done to verify that it is in force and constitutes an efficient means of ensuring and improving the quality of the programs. The main data are, therefore, the files of the program evaluations already realized by the institution.
Educational Evaluation around the World 158
It must be realized that in all these evaluations, the CEEC always pursues its main objective of contributing to the improvement of the quality of college education. This is at the root of the approach it has taken, characterized by support to the institutions through guides, workshops and direct assistance, insistence on efficient self-assessment and use of standards which are for the most part qualitative. Quantitative data are utilized more as a base upon which judgments are made than as absolute norms.
Educational Evaluation around the World 159
France – Higher Education
Bruno Curvale Project Manager
Comité national d'évaluation (CNE), France